2024-09-16T08:42:12.986 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T08:42:13.002 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T08:42:13.085 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi/7906989 branch: wip-guits-main-2024-09-13-1248 description: orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} email: gabrioux@ibm.com first_in_suite: false job_id: '7906989' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-main-2024-09-13-1248 ceph: conf: client: debug ms: 1 global: mon election default strategy: 3 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - CEPHADM_STRAY_DAEMON log-only-match: - CEPHADM_ sha1: 8293d73f8690540e843a81caec373f9cc29cf705 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 8293d73f8690540e843a81caec373f9cc29cf705 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-main-2024-09-13-1248 sha1: 8293d73f8690540e843a81caec373f9cc29cf705 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6714 sha1: 8293d73f8690540e843a81caec373f9cc29cf705 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-main-2024-09-13-1248 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8293d73f8690540e843a81caec373f9cc29cf705/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8293d73f8690540e843a81caec373f9cc29cf705 targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHpXwMWHcrPR0W3+ESESIsd9VVs0Q2mUW1j15Jdw6N/2/RmGhWQWlhcggC9aIrWxk8ynEAQQhcHPOwXx4xZT2PE= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKDa/Ekc+rrMbdZysaHa9Hf5zA7W3p3YmZXIrvfAz5pJPgecmEGvFPqrzOxSzbwHtmBqCR4sQ/NAJVJNOYTM24c= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_07:44:22 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T08:42:13.085 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8293d73f8690540e843a81caec373f9cc29cf705/qa; will attempt to use it 2024-09-16T08:42:13.086 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8293d73f8690540e843a81caec373f9cc29cf705/qa/tasks 2024-09-16T08:42:13.086 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T08:42:13.087 INFO:teuthology.task.internal:Checking packages... 2024-09-16T08:42:13.110 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8293d73f8690540e843a81caec373f9cc29cf705' 2024-09-16T08:42:13.110 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T08:42:13.110 INFO:teuthology.packaging:ref: None 2024-09-16T08:42:13.110 INFO:teuthology.packaging:tag: None 2024-09-16T08:42:13.110 INFO:teuthology.packaging:branch: wip-guits-main-2024-09-13-1248 2024-09-16T08:42:13.110 INFO:teuthology.packaging:sha1: 8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:42:13.111 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-main-2024-09-13-1248 2024-09-16T08:42:13.255 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4917.g8293d73f 2024-09-16T08:42:13.257 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T08:42:13.263 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T08:42:13.263 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T08:42:13.327 INFO:teuthology.task.internal:Saving configuration 2024-09-16T08:42:13.338 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T08:42:13.373 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T08:42:13.398 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi/7906989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 08:36:36.044611', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHpXwMWHcrPR0W3+ESESIsd9VVs0Q2mUW1j15Jdw6N/2/RmGhWQWlhcggC9aIrWxk8ynEAQQhcHPOwXx4xZT2PE='} 2024-09-16T08:42:13.418 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi/7906989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 08:36:36.043416', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKDa/Ekc+rrMbdZysaHa9Hf5zA7W3p3YmZXIrvfAz5pJPgecmEGvFPqrzOxSzbwHtmBqCR4sQ/NAJVJNOYTM24c='} 2024-09-16T08:42:13.418 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T08:42:13.424 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-16T08:42:13.424 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-16T08:42:13.425 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T08:42:13.536 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb134097be0>, signals=[15]) 2024-09-16T08:42:13.536 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T08:42:13.542 INFO:teuthology.task.internal:Opening connections... 2024-09-16T08:42:13.542 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-09-16T08:42:13.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:42:13.617 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T08:42:13.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:42:13.691 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T08:42:13.700 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-09-16T08:42:13.719 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-09-16T08:42:13.719 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-09-16T08:42:13.776 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T08:42:13.777 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T08:42:13.778 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-09-16T08:42:13.795 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-09-16T08:42:13.813 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-09-16T08:42:13.813 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-09-16T08:42:13.871 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-09-16T08:42:13.871 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-09-16T08:42:13.871 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-09-16T08:42:13.871 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-09-16T08:42:13.871 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-09-16T08:42:13.871 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-09-16T08:42:13.871 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T08:42:13.871 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-09-16T08:42:13.872 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-09-16T08:42:13.872 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T08:42:13.872 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-09-16T08:42:13.872 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T08:42:13.872 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T08:42:13.872 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T08:42:13.872 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-09-16T08:42:13.895 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T08:42:13.907 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T08:42:13.935 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T08:42:13.935 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-09-16T08:42:13.938 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-09-16T08:42:13.955 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T08:42:14.037 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T08:42:14.038 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-09-16T08:42:14.040 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-09-16T08:42:14.148 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T08:42:14.205 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T08:42:14.220 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T08:42:14.220 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T08:42:14.220 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T08:42:14.221 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T08:42:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:42:14.238 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-09-16T08:42:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:42:14.239 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-16T08:42:14.296 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-503.el9.x86_64 2024-09-16T08:42:14.297 INFO:teuthology.task.kernel:Running kernel on smithi111: 5.14.0-503.el9.x86_64 2024-09-16T08:42:14.297 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2024-09-16T08:42:14.298 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-503.el9.x86_64 2024-09-16T08:42:14.298 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-503.el9.x86_64 2024-09-16T08:42:14.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-09-16T08:42:16.515 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-16T08:42:17.979 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-09-16T08:42:21.988 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 20 MB 00:02 2024-09-16T08:42:22.374 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 20 MB 00:04 2024-09-16T08:42:26.156 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-16T08:42:27.598 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-16T08:42:28.361 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-16T08:42:28.374 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.5 MB 00:02 2024-09-16T08:42:29.984 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-16T08:42:30.854 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-16T08:42:33.750 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-09-16T08:42:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T08:42:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T08:42:35.857 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:42:35.862 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:42:35.863 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-16T08:42:35.864 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:42:35.864 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 79 M 2024-09-16T08:42:35.864 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 126 M 2024-09-16T08:42:35.864 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:42:36.068 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 220 kB/s | 23 kB 00:00 2024-09-16T08:42:36.319 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T08:42:36.601 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 54 MB/s | 29 MB 00:00 2024-09-16T08:42:37.018 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-09-16T08:42:37.385 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 24 MB/s | 35 MB 00:01 2024-09-16T08:42:37.386 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:42:37.386 INFO:teuthology.orchestra.run.smithi114.stdout:Total 52 MB/s | 79 MB 00:01 2024-09-16T08:42:38.067 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:42:38.317 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T08:42:38.317 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T08:42:38.410 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:42:38.420 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:42:38.421 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:42:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:42:38.421 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-09-16T08:42:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout:Install 4 Packages 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 79 M 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 126 M 2024-09-16T08:42:38.422 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:42:38.779 INFO:teuthology.orchestra.run.smithi111.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 291 kB/s | 23 kB 00:00 2024-09-16T08:42:39.396 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:42:39.397 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:42:41.831 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:42:42.509 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T08:42:42.594 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T08:42:42.962 INFO:teuthology.orchestra.run.smithi111.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.7 MB/s | 16 MB 00:04 2024-09-16T08:42:45.401 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T08:42:45.545 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T08:42:46.161 INFO:teuthology.orchestra.run.smithi111.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 3.9 MB/s | 29 MB 00:07 2024-09-16T08:42:47.912 INFO:teuthology.orchestra.run.smithi111.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 3.7 MB/s | 35 MB 00:09 2024-09-16T08:42:47.912 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:42:47.912 INFO:teuthology.orchestra.run.smithi111.stdout:Total 8.3 MB/s | 79 MB 00:09 2024-09-16T08:42:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:42:48.939 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:42:48.939 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:42:49.919 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:42:49.919 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:42:50.380 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T08:42:50.421 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:42:52.365 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:42:53.026 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T08:42:53.112 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T08:42:55.276 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:42:55.930 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T08:42:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T08:43:00.888 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T08:43:00.923 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:05.746 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:33.597 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:33.637 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T08:43:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T08:43:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T08:43:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:43:34.917 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:43:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T08:43:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T08:43:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T08:43:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T08:43:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:43:34.918 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:43:35.261 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T08:43:35.351 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T08:43:35.352 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T08:43:35.352 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T08:43:35.352 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T08:43:35.352 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T08:43:35.352 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T08:43:35.352 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T08:43:35.352 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T08:43:35.411 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T08:43:35.412 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T08:43:35.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-09-16T08:43:35.444 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-09-16T08:43:35.444 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-09-16T08:43:35.444 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-09-16T08:43:35.445 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 82.el9 2024-09-16T08:43:35.445 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-09-16T08:43:35.445 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T08:43:35.445 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-09-16T08:43:35.445 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8274670 2024-09-16T08:43:35.445 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-09-16T08:43:35.445 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T08:43:35.445 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T08:43:35.445 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T08:43:35.446 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T08:43:35.446 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-09-16T08:43:35.446 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-09-16T08:43:35.446 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T08:43:35.446 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-09-16T08:43:35.446 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-09-16T08:43:35.446 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:43:35.446 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T08:43:35.446 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T08:43:35.447 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T08:43:35.447 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-09-16T08:43:35.447 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:43:35.447 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T08:43:35.448 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T08:43:35.448 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-505.el9.x86_64 2024-09-16T08:43:35.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T08:43:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-09-16T08:43:37.455 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T08:43:37.485 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-09-16T08:43:37.488 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-16T08:43:37.503 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.5Ug96tozOd 2024-09-16T08:43:37.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5Ug96tozOd 2024-09-16T08:43:37.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.5Ug96tozOd 2024-09-16T08:43:37.709 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.5Ug96tozOd is 6KB 2024-09-16T08:43:37.729 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.5Ug96tozOd 2024-09-16T08:43:37.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T08:43:37.811 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T08:43:37.812 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T08:43:37.812 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T08:43:37.812 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T08:43:37.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T08:43:37.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-09-16T08:43:44.218 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:44.267 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:44.642 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:44.642 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T08:43:44.642 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T08:43:44.642 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T08:43:45.515 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T08:43:45.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:43:45.515 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:43:45.515 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T08:43:45.515 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T08:43:45.516 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T08:43:45.516 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T08:43:45.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:43:45.516 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:43:45.893 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T08:43:45.981 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T08:43:45.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T08:43:45.981 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T08:43:45.981 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T08:43:45.981 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T08:43:45.982 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T08:43:45.982 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T08:43:45.982 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T08:43:46.040 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T08:43:46.040 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T08:43:46.040 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2024-09-16T08:43:46.072 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2024-09-16T08:43:46.072 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.06 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 82.el9 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Group : Unspecified 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 8274670 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : builder@centos.org 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : CentOS 2024-09-16T08:43:46.073 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T08:43:46.074 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2024-09-16T08:43:46.074 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2024-09-16T08:43:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:43:46.074 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T08:43:46.074 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T08:43:46.074 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T08:43:46.074 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2024-09-16T08:43:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:43:46.074 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T08:43:46.075 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T08:43:46.075 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 5.14.0-505.el9.x86_64 2024-09-16T08:43:46.075 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T08:43:46.724 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2024-09-16T08:43:47.989 INFO:teuthology.orchestra.run.smithi111.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T08:43:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:done 2024-09-16T08:43:48.029 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-09-16T08:43:48.044 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.HNzPek2pio 2024-09-16T08:43:48.045 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HNzPek2pio 2024-09-16T08:43:48.111 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.HNzPek2pio 2024-09-16T08:43:48.258 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.HNzPek2pio is 6KB 2024-09-16T08:43:48.308 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.HNzPek2pio 2024-09-16T08:43:48.322 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T08:43:48.385 INFO:teuthology.orchestra.run.smithi111.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T08:43:48.386 INFO:teuthology.orchestra.run.smithi111.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T08:43:48.386 INFO:teuthology.orchestra.run.smithi111.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T08:43:48.386 INFO:teuthology.orchestra.run.smithi111.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T08:43:48.387 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T08:43:48.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2024-09-16T08:44:08.004 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T08:44:08.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T08:44:08.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:44:18.555 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T08:44:18.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-09-16T08:44:18.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:44:53.186 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-09-16T08:45:02.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-09-16T08:45:02.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:45:05.245 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-09-16T08:45:08.009 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T08:45:17.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T08:45:17.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:45:17.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-09-16T08:45:17.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:45:18.486 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-16T08:45:18.909 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T08:45:18.910 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-16T08:45:18.910 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-16T08:45:18.925 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-505.el9.x86_64 2024-09-16T08:45:18.926 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T08:45:18.926 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T08:45:18.926 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T08:45:19.927 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T08:45:19.927 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T08:45:20.000 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-09-16T08:45:20.025 DEBUG:teuthology.parallel:result is None 2024-09-16T08:45:35.677 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-09-16T08:45:50.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-09-16T08:45:50.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:45:50.990 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-09-16T08:45:51.505 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-09-16T08:45:51.505 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T08:45:51.505 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-09-16T08:45:51.545 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-505.el9.x86_64 2024-09-16T08:45:51.545 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T08:45:51.545 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T08:45:51.545 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T08:45:52.546 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T08:45:52.547 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T08:45:52.609 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2024-09-16T08:45:52.640 DEBUG:teuthology.parallel:result is None 2024-09-16T08:45:52.640 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T08:45:52.655 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T08:45:52.656 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T08:45:52.658 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T08:45:52.679 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T08:45:52.705 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T08:45:52.746 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T08:45:52.747 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T08:45:52.749 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T08:45:52.936 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T08:45:52.947 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T08:45:52.947 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T08:45:52.950 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T08:45:53.006 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:45:53.018 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:45:53.040 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:45:53.051 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T08:45:53.052 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T08:45:53.060 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T08:45:53.061 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T08:45:53.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T08:45:53.091 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T08:45:53.117 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T08:45:53.118 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T08:45:53.130 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T08:45:53.147 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T08:45:53.227 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T08:45:53.275 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T08:45:53.358 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T08:45:53.398 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:45:53.398 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T08:45:53.465 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T08:45:53.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T08:45:53.571 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T08:45:53.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T08:45:53.655 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:45:53.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T08:45:53.720 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-09-16T08:45:53.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-09-16T08:45:53.765 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T08:45:53.810 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T08:45:54.210 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T08:45:54.218 INFO:teuthology.task.internal:Starting timer... 2024-09-16T08:45:54.218 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T08:45:54.259 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T08:45:54.334 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-16T08:45:54.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-09-16T08:45:54.402 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-09-16T08:45:54.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-09-16T08:45:54.469 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-09-16T08:45:54.471 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T08:45:54.471 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-09-16T08:45:54.498 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-09-16T08:45:54.498 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-09-16T08:45:54.520 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-09-16T08:45:54.521 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive'} 2024-09-16T08:45:54.521 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-16T08:45:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:45:54.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-16T08:45:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:45:54.590 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T08:45:54.590 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-09-16T08:45:54.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-09-16T08:45:54.671 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T08:45:54.682 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T08:45:54.684 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T08:45:54.833 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T08:45:54.844 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T08:45:54.845 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi111.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T08:49:47.802 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com')] 2024-09-16T08:49:47.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-09-16T08:49:47.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:49:47.881 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-09-16T08:49:47.960 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-09-16T08:49:47.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T08:49:47.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T08:49:48.036 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-16T08:49:48.070 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-16T08:49:48.070 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T08:49:48.081 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T08:49:48.081 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T08:49:48.082 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T08:49:48.085 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T08:49:48.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T08:49:48.113 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T08:49:48.128 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T08:49:48.139 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T08:49:48.152 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-09-16T08:49:48.153 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T08:49:48.163 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-09-16T08:49:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-09-16T08:49:48.178 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T08:49:48.188 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-09-16T08:49:48.192 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T08:49:48.202 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T08:49:48.216 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T08:49:48.254 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-09-16T08:49:48.257 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T08:49:48.257 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-09-16T08:49:48.257 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:49:48.257 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:49:48.257 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:49:48.257 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:49:48.275 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-16T08:49:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T08:49:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-16T08:49:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:49:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:49:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:49:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T08:49:48.279 INFO:teuthology.run_tasks:Running task pexec... 2024-09-16T08:49:48.288 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-16T08:49:48.288 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi111.front.sepia.ceph.com 2024-09-16T08:49:48.289 DEBUG:teuthology.orchestra.run.smithi111:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-16T08:49:48.289 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T08:49:48.289 DEBUG:teuthology.orchestra.run.smithi114:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-16T08:49:48.645 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:49:48.653 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout:Removing: 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout:Removing unused dependencies: 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout:Remove 2 Packages 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:49:48.654 INFO:teuthology.orchestra.run.smithi111.stdout:Freed space: 5.5 M 2024-09-16T08:49:48.655 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:49:48.658 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:49:48.658 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:49:48.667 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:49:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:49:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-16T08:49:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:49:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:Removing: 2024-09-16T08:49:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-16T08:49:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:Removing unused dependencies: 2024-09-16T08:49:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-16T08:49:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:49:48.686 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:49:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:49:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:Remove 2 Packages 2024-09-16T08:49:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:49:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:Freed space: 5.5 M 2024-09-16T08:49:48.687 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:49:48.690 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:49:48.690 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:49:48.721 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:49:48.722 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:49:48.755 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:49:48.755 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:49:48.865 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:49:48.913 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:49:48.945 INFO:teuthology.orchestra.run.smithi111.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-16T08:49:49.004 INFO:teuthology.orchestra.run.smithi114.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-16T08:49:49.004 INFO:teuthology.orchestra.run.smithi111.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-16T08:49:49.063 INFO:teuthology.orchestra.run.smithi114.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-16T08:49:49.515 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-16T08:49:49.515 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-16T08:49:49.565 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-16T08:49:49.565 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-16T08:49:49.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-16T08:49:49.781 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:49:49.781 INFO:teuthology.orchestra.run.smithi111.stdout:Removed: 2024-09-16T08:49:49.781 INFO:teuthology.orchestra.run.smithi111.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-16T08:49:49.781 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:49:49.781 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:49:49.786 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-16T08:49:49.786 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:49:49.786 INFO:teuthology.orchestra.run.smithi114.stdout:Removed: 2024-09-16T08:49:49.786 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-16T08:49:49.786 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:49:49.786 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:49:50.594 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 08:48:53 AM UTC. 2024-09-16T08:49:50.599 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:49 ago on Mon 16 Sep 2024 08:49:01 AM UTC. 2024-09-16T08:49:50.729 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:49:50.730 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:49:50.730 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:49:50.730 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:49:50.730 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:49:50.730 INFO:teuthology.orchestra.run.smithi111.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout:Install 7 Packages 2024-09-16T08:49:50.731 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:49:50.746 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:49:50.747 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:49:50.747 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:49:50.747 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout:Install 7 Packages 2024-09-16T08:49:50.748 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:49:50.749 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 5.2 M 2024-09-16T08:49:50.749 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 19 M 2024-09-16T08:49:50.749 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:49:50.782 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 5.2 M 2024-09-16T08:49:50.782 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 19 M 2024-09-16T08:49:50.782 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:49:50.919 INFO:teuthology.orchestra.run.smithi114.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 477 kB/s | 44 kB 00:00 2024-09-16T08:49:50.944 INFO:teuthology.orchestra.run.smithi114.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 893 kB/s | 106 kB 00:00 2024-09-16T08:49:50.981 INFO:teuthology.orchestra.run.smithi114.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 1.2 MB/s | 72 kB 00:00 2024-09-16T08:49:51.004 INFO:teuthology.orchestra.run.smithi114.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-16T08:49:51.037 INFO:teuthology.orchestra.run.smithi114.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.2 MB/s | 906 kB 00:00 2024-09-16T08:49:51.087 INFO:teuthology.orchestra.run.smithi114.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 7.5 MB/s | 837 kB 00:00 2024-09-16T08:49:51.404 INFO:teuthology.orchestra.run.smithi114.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 7.9 MB/s | 3.2 MB 00:00 2024-09-16T08:49:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:49:51.405 INFO:teuthology.orchestra.run.smithi114.stdout:Total 8.3 MB/s | 5.2 MB 00:00 2024-09-16T08:49:51.515 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:49:51.530 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:49:51.530 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:49:51.686 INFO:teuthology.orchestra.run.smithi111.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 70 kB/s | 44 kB 00:00 2024-09-16T08:49:51.785 INFO:teuthology.orchestra.run.smithi111.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 144 kB/s | 106 kB 00:00 2024-09-16T08:49:51.838 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:49:51.839 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:49:52.019 INFO:teuthology.orchestra.run.smithi111.stdout:(3/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 934 kB/s | 906 kB 00:00 2024-09-16T08:49:52.317 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:49:52.361 INFO:teuthology.orchestra.run.smithi111.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 106 kB/s | 72 kB 00:00 2024-09-16T08:49:52.406 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-16T08:49:52.442 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-16T08:49:52.453 INFO:teuthology.orchestra.run.smithi111.stdout:(5/7): python3-kmod-0.9-32.el9.x86_64.rpm 126 kB/s | 84 kB 00:00 2024-09-16T08:49:52.470 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-16T08:49:52.675 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-16T08:49:52.711 INFO:teuthology.orchestra.run.smithi111.stdout:(6/7): runc-1.1.13-4.el9.x86_64.rpm 9.1 MB/s | 3.2 MB 00:00 2024-09-16T08:49:52.717 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-16T08:49:52.996 INFO:teuthology.orchestra.run.smithi111.stdout:(7/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 866 kB/s | 837 kB 00:00 2024-09-16T08:49:52.996 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:49:52.996 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.3 MB/s | 5.2 MB 00:02 2024-09-16T08:49:53.113 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:49:53.128 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:49:53.128 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:49:53.166 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-16T08:49:53.204 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-16T08:49:53.313 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-16T08:49:53.375 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-16T08:49:53.402 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:49:53.403 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:49:53.861 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:49:53.948 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-16T08:49:53.982 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-16T08:49:54.011 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-16T08:49:54.219 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-16T08:49:54.250 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-16T08:49:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-16T08:49:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-16T08:49:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-16T08:49:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-16T08:49:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-16T08:49:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-16T08:49:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-16T08:49:54.695 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-16T08:49:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-16T08:49:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:49:54.698 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:49:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-16T08:49:54.698 INFO:teuthology.orchestra.run.smithi114.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-16T08:49:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-16T08:49:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-16T08:49:54.699 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:49:54.699 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:49:54.719 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-16T08:49:54.834 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-16T08:49:54.915 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-16T08:49:55.075 DEBUG:teuthology.parallel:result is None 2024-09-16T08:49:55.940 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-16T08:49:55.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-16T08:49:55.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-16T08:49:55.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-16T08:49:55.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-16T08:49:55.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-16T08:49:55.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-16T08:49:56.335 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-16T08:49:56.335 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:49:56.335 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:49:56.335 INFO:teuthology.orchestra.run.smithi111.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-16T08:49:56.335 INFO:teuthology.orchestra.run.smithi111.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-16T08:49:56.335 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-16T08:49:56.335 INFO:teuthology.orchestra.run.smithi111.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-16T08:49:56.335 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:49:56.336 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:49:56.748 DEBUG:teuthology.parallel:result is None 2024-09-16T08:49:56.748 INFO:teuthology.run_tasks:Running task install... 2024-09-16T08:49:56.758 DEBUG:teuthology.task.install:project ceph 2024-09-16T08:49:56.758 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8293d73f8690540e843a81caec373f9cc29cf705'}} 2024-09-16T08:49:56.758 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8293d73f8690540e843a81caec373f9cc29cf705'} 2024-09-16T08:49:56.759 INFO:teuthology.task.install:Using flavor: default 2024-09-16T08:49:56.764 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T08:49:56.765 INFO:teuthology.task.install:extra packages: [] 2024-09-16T08:49:56.765 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8293d73f8690540e843a81caec373f9cc29cf705', 'tag': None, 'wait_for_package': False} 2024-09-16T08:49:56.766 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:49:56.768 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8293d73f8690540e843a81caec373f9cc29cf705', 'tag': None, 'wait_for_package': False} 2024-09-16T08:49:56.768 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:49:56.912 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/ 2024-09-16T08:49:56.912 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4917.g8293d73f 2024-09-16T08:49:56.915 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/ 2024-09-16T08:49:56.916 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4917.g8293d73f 2024-09-16T08:49:57.027 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T08:49:57.028 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:49:57.028 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T08:49:57.031 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T08:49:57.031 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:49:57.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T08:49:57.054 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T08:49:57.054 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8293d73f8690540e843a81caec373f9cc29cf705/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T08:49:57.063 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T08:49:57.063 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8293d73f8690540e843a81caec373f9cc29cf705/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T08:49:57.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T08:49:57.145 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T08:49:57.179 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T08:49:57.213 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T08:49:57.243 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-09-16T08:49:57.245 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-09-16T08:49:57.280 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-09-16T08:49:57.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-09-16T08:49:57.470 INFO:teuthology.orchestra.run.smithi111.stdout:45 files removed 2024-09-16T08:49:57.497 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw 2024-09-16T08:49:57.504 INFO:teuthology.orchestra.run.smithi114.stdout:45 files removed 2024-09-16T08:49:57.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-09-16T08:49:58.054 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 309 kB/s | 84 kB 00:00 2024-09-16T08:49:58.097 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 299 kB/s | 84 kB 00:00 2024-09-16T08:49:58.338 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-16T08:49:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 89 kB/s | 24 kB 00:00 2024-09-16T08:49:58.613 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-16T08:49:58.690 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-16T08:50:00.614 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.2 MB 00:01 2024-09-16T08:50:01.631 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.2 MB 00:02 2024-09-16T08:50:02.648 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-16T08:50:03.669 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-16T08:50:08.201 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-16T08:50:08.974 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-09-16T08:50:10.077 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 19 kB 00:00 2024-09-16T08:50:10.616 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 19 kB 00:00 2024-09-16T08:50:11.247 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-16T08:50:11.504 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-16T08:50:16.687 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-16T08:50:16.984 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-09-16T08:50:19.063 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:50:19.065 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:50:19.065 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:50:19.065 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:50:19.065 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:50:19.065 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 11 M 2024-09-16T08:50:19.065 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 3.5 M 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 3.2 M 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 5.5 M 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 22 M 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 25 k 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 755 k 2024-09-16T08:50:19.066 INFO:teuthology.orchestra.run.smithi114.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 508 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 5.5 M 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 45 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 143 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 161 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T08:50:19.067 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 320 k 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 306 k 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 99 k 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout:Install 31 Packages 2024-09-16T08:50:19.068 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-09-16T08:50:19.069 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:19.069 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 62 M 2024-09-16T08:50:19.069 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:50:19.390 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:50:19.392 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:50:19.392 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:50:19.392 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:50:19.392 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:50:19.392 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 11 M 2024-09-16T08:50:19.392 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-09-16T08:50:19.392 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 3.5 M 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 3.2 M 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 5.5 M 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 22 M 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 25 k 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 755 k 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T08:50:19.393 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 508 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 5.5 M 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 45 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 143 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 161 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 320 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 306 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 99 k 2024-09-16T08:50:19.394 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout:Install 31 Packages 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-09-16T08:50:19.395 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:19.396 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 62 M 2024-09-16T08:50:19.396 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:50:19.821 INFO:teuthology.orchestra.run.smithi114.stdout:(1/33): ceph-base-19.3.0-4917.g8293d73f.el9.x86 9.6 MB/s | 5.5 MB 00:00 2024-09-16T08:50:19.872 INFO:teuthology.orchestra.run.smithi114.stdout:(2/33): ceph-selinux-19.3.0-4917.g8293d73f.el9. 504 kB/s | 25 kB 00:00 2024-09-16T08:50:19.963 INFO:teuthology.orchestra.run.smithi114.stdout:(3/33): libcephfs2-19.3.0-4917.g8293d73f.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-16T08:50:20.080 INFO:teuthology.orchestra.run.smithi114.stdout:(4/33): ceph-radosgw-19.3.0-4917.g8293d73f.el9. 13 MB/s | 11 MB 00:00 2024-09-16T08:50:20.114 INFO:teuthology.orchestra.run.smithi114.stdout:(5/33): libradosstriper1-19.3.0-4917.g8293d73f. 3.3 MB/s | 508 kB 00:00 2024-09-16T08:50:20.181 INFO:teuthology.orchestra.run.smithi114.stdout:(6/33): python3-ceph-argparse-19.3.0-4917.g8293 678 kB/s | 45 kB 00:00 2024-09-16T08:50:20.281 INFO:teuthology.orchestra.run.smithi114.stdout:(7/33): python3-ceph-common-19.3.0-4917.g8293d7 1.4 MB/s | 143 kB 00:00 2024-09-16T08:50:20.381 INFO:teuthology.orchestra.run.smithi114.stdout:(8/33): python3-cephfs-19.3.0-4917.g8293d73f.el 1.6 MB/s | 161 kB 00:00 2024-09-16T08:50:20.490 INFO:teuthology.orchestra.run.smithi114.stdout:(9/33): librgw2-19.3.0-4917.g8293d73f.el9.x86_6 13 MB/s | 5.5 MB 00:00 2024-09-16T08:50:20.515 INFO:teuthology.orchestra.run.smithi114.stdout:(10/33): python3-rados-19.3.0-4917.g8293d73f.el 2.3 MB/s | 320 kB 00:00 2024-09-16T08:50:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:(11/33): ceph-common-19.3.0-4917.g8293d73f.el9. 15 MB/s | 22 MB 00:01 2024-09-16T08:50:20.749 INFO:teuthology.orchestra.run.smithi114.stdout:(12/33): python3-rgw-19.3.0-4917.g8293d73f.el9. 425 kB/s | 99 kB 00:00 2024-09-16T08:50:20.774 INFO:teuthology.orchestra.run.smithi114.stdout:(13/33): python3-rbd-19.3.0-4917.g8293d73f.el9. 1.1 MB/s | 306 kB 00:00 2024-09-16T08:50:20.785 INFO:teuthology.orchestra.run.smithi111.stdout:(1/33): ceph-base-19.3.0-4917.g8293d73f.el9.x86 5.9 MB/s | 5.5 MB 00:00 2024-09-16T08:50:20.853 INFO:teuthology.orchestra.run.smithi111.stdout:(2/33): ceph-selinux-19.3.0-4917.g8293d73f.el9. 378 kB/s | 25 kB 00:00 2024-09-16T08:50:20.969 INFO:teuthology.orchestra.run.smithi111.stdout:(3/33): ceph-radosgw-19.3.0-4917.g8293d73f.el9. 9.9 MB/s | 11 MB 00:01 2024-09-16T08:50:21.011 INFO:teuthology.orchestra.run.smithi111.stdout:(4/33): libcephfs2-19.3.0-4917.g8293d73f.el9.x8 4.7 MB/s | 755 kB 00:00 2024-09-16T08:50:21.037 INFO:teuthology.orchestra.run.smithi111.stdout:(5/33): libradosstriper1-19.3.0-4917.g8293d73f. 7.4 MB/s | 508 kB 00:00 2024-09-16T08:50:21.087 INFO:teuthology.orchestra.run.smithi111.stdout:(6/33): python3-ceph-argparse-19.3.0-4917.g8293 905 kB/s | 45 kB 00:00 2024-09-16T08:50:21.321 INFO:teuthology.orchestra.run.smithi111.stdout:(7/33): ceph-common-19.3.0-4917.g8293d73f.el9.x 15 MB/s | 22 MB 00:01 2024-09-16T08:50:21.346 INFO:teuthology.orchestra.run.smithi111.stdout:(8/33): python3-ceph-common-19.3.0-4917.g8293d7 554 kB/s | 143 kB 00:00 2024-09-16T08:50:21.371 INFO:teuthology.orchestra.run.smithi111.stdout:(9/33): python3-cephfs-19.3.0-4917.g8293d73f.el 3.2 MB/s | 161 kB 00:00 2024-09-16T08:50:21.405 INFO:teuthology.orchestra.run.smithi111.stdout:(10/33): python3-rados-19.3.0-4917.g8293d73f.el 5.3 MB/s | 320 kB 00:00 2024-09-16T08:50:21.430 INFO:teuthology.orchestra.run.smithi111.stdout:(11/33): python3-rbd-19.3.0-4917.g8293d73f.el9. 5.1 MB/s | 306 kB 00:00 2024-09-16T08:50:21.433 INFO:teuthology.orchestra.run.smithi114.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 151 kB/s | 107 kB 00:00 2024-09-16T08:50:21.455 INFO:teuthology.orchestra.run.smithi111.stdout:(12/33): python3-rgw-19.3.0-4917.g8293d73f.el9. 1.9 MB/s | 99 kB 00:00 2024-09-16T08:50:21.468 INFO:teuthology.orchestra.run.smithi114.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 238 kB/s | 171 kB 00:00 2024-09-16T08:50:21.533 INFO:teuthology.orchestra.run.smithi114.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 211 kB/s | 160 kB 00:00 2024-09-16T08:50:21.539 INFO:teuthology.orchestra.run.smithi111.stdout:(13/33): librgw2-19.3.0-4917.g8293d73f.el9.x86_ 10 MB/s | 5.5 MB 00:00 2024-09-16T08:50:21.589 INFO:teuthology.orchestra.run.smithi111.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 673 kB/s | 107 kB 00:00 2024-09-16T08:50:21.631 INFO:teuthology.orchestra.run.smithi111.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-16T08:50:21.656 INFO:teuthology.orchestra.run.smithi111.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 850 kB/s | 171 kB 00:00 2024-09-16T08:50:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-09-16T08:50:21.707 INFO:teuthology.orchestra.run.smithi111.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-09-16T08:50:21.732 INFO:teuthology.orchestra.run.smithi111.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-09-16T08:50:21.757 INFO:teuthology.orchestra.run.smithi111.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-16T08:50:21.783 INFO:teuthology.orchestra.run.smithi111.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 715 kB/s | 36 kB 00:00 2024-09-16T08:50:21.808 INFO:teuthology.orchestra.run.smithi111.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-09-16T08:50:21.833 INFO:teuthology.orchestra.run.smithi111.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-16T08:50:21.892 INFO:teuthology.orchestra.run.smithi111.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-16T08:50:21.917 INFO:teuthology.orchestra.run.smithi111.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 371 kB/s | 26 kB 00:00 2024-09-16T08:50:21.942 INFO:teuthology.orchestra.run.smithi111.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-09-16T08:50:21.968 INFO:teuthology.orchestra.run.smithi111.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T08:50:22.010 INFO:teuthology.orchestra.run.smithi111.stdout:(28/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-16T08:50:22.035 INFO:teuthology.orchestra.run.smithi111.stdout:(29/33): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-16T08:50:22.060 INFO:teuthology.orchestra.run.smithi114.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-09-16T08:50:22.144 INFO:teuthology.orchestra.run.smithi111.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-16T08:50:22.219 INFO:teuthology.orchestra.run.smithi111.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-16T08:50:22.310 INFO:teuthology.orchestra.run.smithi114.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 377 kB/s | 292 kB 00:00 2024-09-16T08:50:22.311 INFO:teuthology.orchestra.run.smithi111.stdout:(32/33): librados2-19.3.0-4917.g8293d73f.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-16T08:50:22.343 INFO:teuthology.orchestra.run.smithi114.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 756 kB/s | 662 kB 00:00 2024-09-16T08:50:22.403 INFO:teuthology.orchestra.run.smithi111.stdout:(33/33): librbd1-19.3.0-4917.g8293d73f.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-16T08:50:22.408 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:50:22.408 INFO:teuthology.orchestra.run.smithi111.stdout:Total 21 MB/s | 62 MB 00:03 2024-09-16T08:50:22.520 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:50:22.557 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:50:22.557 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:50:22.668 INFO:teuthology.orchestra.run.smithi114.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 69 kB/s | 42 kB 00:00 2024-09-16T08:50:22.902 INFO:teuthology.orchestra.run.smithi114.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 64 kB/s | 36 kB 00:00 2024-09-16T08:50:22.961 INFO:teuthology.orchestra.run.smithi114.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-16T08:50:23.073 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:50:23.074 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:50:23.103 INFO:teuthology.orchestra.run.smithi114.stdout:(23/33): libarrow-9.0.0-11.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-09-16T08:50:23.128 INFO:teuthology.orchestra.run.smithi114.stdout:(24/33): qatlib-24.02.0-1.el9.x86_64.rpm 272 kB/s | 221 kB 00:00 2024-09-16T08:50:23.145 INFO:teuthology.orchestra.run.smithi114.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 624 kB/s | 26 kB 00:00 2024-09-16T08:50:23.187 INFO:teuthology.orchestra.run.smithi114.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.9 MB/s | 67 kB 00:00 2024-09-16T08:50:23.212 INFO:teuthology.orchestra.run.smithi114.stdout:(27/33): liboath-2.6.7-2.el9.x86_64.rpm 580 kB/s | 49 kB 00:00 2024-09-16T08:50:23.246 INFO:teuthology.orchestra.run.smithi114.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T08:50:23.279 INFO:teuthology.orchestra.run.smithi114.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.1 MB/s | 839 kB 00:00 2024-09-16T08:50:23.321 INFO:teuthology.orchestra.run.smithi114.stdout:(30/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 73 kB/s | 47 kB 00:00 2024-09-16T08:50:23.372 INFO:teuthology.orchestra.run.smithi114.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-16T08:50:23.539 INFO:teuthology.orchestra.run.smithi114.stdout:(32/33): librados2-19.3.0-4917.g8293d73f.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-16T08:50:23.622 INFO:teuthology.orchestra.run.smithi114.stdout:(33/33): librbd1-19.3.0-4917.g8293d73f.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-16T08:50:23.627 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:50:23.627 INFO:teuthology.orchestra.run.smithi114.stdout:Total 14 MB/s | 62 MB 00:04 2024-09-16T08:50:23.740 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:50:23.778 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:50:23.778 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:50:24.275 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:50:24.276 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:50:24.590 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:50:24.671 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T08:50:24.723 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T08:50:24.758 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T08:50:24.781 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T08:50:24.806 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T08:50:25.001 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T08:50:25.053 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T08:50:25.414 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T08:50:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 7/35 2024-09-16T08:50:25.628 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 7/35 2024-09-16T08:50:25.718 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:19.3.0-4917.g8293d73f.el9.x86_64 8/35 2024-09-16T08:50:25.745 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 9/35 2024-09-16T08:50:25.753 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:50:25.808 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 9/35 2024-09-16T08:50:25.837 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T08:50:25.845 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T08:50:25.877 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T08:50:25.880 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:19.3.0-4917.g8293d73f.el9. 11/35 2024-09-16T08:50:25.902 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T08:50:25.925 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T08:50:25.945 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T08:50:25.951 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:19.3.0-4917.g8293d73f.el9.x86_64 12/35 2024-09-16T08:50:25.981 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:19.3.0-4917.g8293d73f.el9.x86_6 13/35 2024-09-16T08:50:26.045 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4917.g8293d73f.el9.x86_6 13/35 2024-09-16T08:50:26.079 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T08:50:26.143 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T08:50:26.188 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T08:50:26.197 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T08:50:26.222 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T08:50:26.270 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T08:50:26.304 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T08:50:26.332 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T08:50:26.366 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T08:50:26.403 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T08:50:26.584 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T08:50:26.654 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 7/35 2024-09-16T08:50:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T08:50:26.746 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 7/35 2024-09-16T08:50:26.771 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 23/35 2024-09-16T08:50:26.837 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.3.0-4917.g8293d73f.el9.x86_64 8/35 2024-09-16T08:50:26.854 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 23/35 2024-09-16T08:50:26.883 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 9/35 2024-09-16T08:50:26.895 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:19.3.0-4917.g8293d73f.el9.x86_64 24/35 2024-09-16T08:50:26.937 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T08:50:26.954 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 9/35 2024-09-16T08:50:26.993 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T08:50:27.043 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.3.0-4917.g8293d73f.el9. 11/35 2024-09-16T08:50:27.134 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.3.0-4917.g8293d73f.el9.x86_64 12/35 2024-09-16T08:50:27.166 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.3.0-4917.g8293d73f.el9.x86_6 13/35 2024-09-16T08:50:27.233 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4917.g8293d73f.el9.x86_6 13/35 2024-09-16T08:50:27.268 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T08:50:27.397 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T08:50:27.432 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T08:50:27.449 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T08:50:27.473 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T08:50:27.474 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:19.3.0-4917.g8293d73f.el9.x86_64 27/35 2024-09-16T08:50:27.514 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T08:50:27.536 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:19.3.0-4917.g8293d73f.el9.x86_64 27/35 2024-09-16T08:50:27.560 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T08:50:27.615 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:19.3.0-4917.g8293d73f.el9.x86_64 28/35 2024-09-16T08:50:27.620 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T08:50:27.643 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:19.3.0-4917.g8293d73f.el9.x8 29/35 2024-09-16T08:50:27.655 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T08:50:27.982 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T08:50:28.023 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 23/35 2024-09-16T08:50:28.114 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 23/35 2024-09-16T08:50:28.145 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.3.0-4917.g8293d73f.el9.x86_64 24/35 2024-09-16T08:50:28.180 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T08:50:28.675 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T08:50:28.700 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.3.0-4917.g8293d73f.el9.x86_64 27/35 2024-09-16T08:50:28.772 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.3.0-4917.g8293d73f.el9.x86_64 27/35 2024-09-16T08:50:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.3.0-4917.g8293d73f.el9.x86_64 28/35 2024-09-16T08:50:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.3.0-4917.g8293d73f.el9.x8 29/35 2024-09-16T08:50:29.988 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 30/35 2024-09-16T08:50:30.023 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 30/35 2024-09-16T08:50:30.603 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 30/35 2024-09-16T08:50:30.642 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 31/35 2024-09-16T08:50:30.716 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 31/35 2024-09-16T08:50:30.716 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T08:50:30.717 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T08:50:30.717 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:30.760 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:19.3.0-4917.g8293d73f.el9.x86_64 32/35 2024-09-16T08:50:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 30/35 2024-09-16T08:50:31.291 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 30/35 2024-09-16T08:50:31.875 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 30/35 2024-09-16T08:50:31.920 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 31/35 2024-09-16T08:50:32.001 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 31/35 2024-09-16T08:50:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T08:50:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T08:50:32.001 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:32.046 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.3.0-4917.g8293d73f.el9.x86_64 32/35 2024-09-16T08:50:40.706 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4917.g8293d73f.el9.x86_64 32/35 2024-09-16T08:50:40.706 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-09-16T08:50:40.706 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-09-16T08:50:40.706 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-09-16T08:50:40.706 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-09-16T08:50:40.706 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-09-16T08:50:40.706 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /root 2024-09-16T08:50:40.706 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-09-16T08:50:40.706 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:41.734 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:19.3.0-4917.g8293d73f.el9.x86_64 33/35 2024-09-16T08:50:41.760 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4917.g8293d73f.el9.x86_64 33/35 2024-09-16T08:50:41.760 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:50:41.761 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T08:50:41.761 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T08:50:41.761 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T08:50:41.761 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:41.762 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T08:50:41.846 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T08:50:41.846 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T08:50:42.281 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4917.g8293d73f.el9.x86_64 32/35 2024-09-16T08:50:42.281 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-09-16T08:50:42.281 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-09-16T08:50:42.281 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-09-16T08:50:42.281 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-09-16T08:50:42.282 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-09-16T08:50:42.282 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-09-16T08:50:42.282 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-09-16T08:50:42.282 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:43.044 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T08:50:43.044 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 1/35 2024-09-16T08:50:43.044 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 2/35 2024-09-16T08:50:43.044 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:19.3.0-4917.g8293d73f.el9.x86_64 3/35 2024-09-16T08:50:43.044 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:19.3.0-4917.g8293d73f.el9.x86_64 4/35 2024-09-16T08:50:43.044 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 5/35 2024-09-16T08:50:43.044 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:19.3.0-4917.g8293d73f.el9.x86_6 6/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:19.3.0-4917.g8293d73f.el9.x86_64 7/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4917.g8293d73f.el9. 8/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:19.3.0-4917.g8293d73f.el9.x8 9/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:19.3.0-4917.g8293d73f.el9.x86_64 10/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:19.3.0-4917.g8293d73f.el9.x86_64 11/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:19.3.0-4917.g8293d73f.el9.x86_64 12/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:19.3.0-4917.g8293d73f.el9.x86_64 13/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T08:50:43.045 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T08:50:43.046 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T08:50:43.046 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T08:50:43.046 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T08:50:43.046 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T08:50:43.046 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T08:50:43.046 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T08:50:43.046 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T08:50:43.046 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T08:50:43.046 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T08:50:43.047 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T08:50:43.047 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T08:50:43.047 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T08:50:43.048 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T08:50:43.048 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 32/35 2024-09-16T08:50:43.048 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T08:50:43.048 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 34/35 2024-09-16T08:50:43.307 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.3.0-4917.g8293d73f.el9.x86_64 33/35 2024-09-16T08:50:43.333 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4917.g8293d73f.el9.x86_64 33/35 2024-09-16T08:50:43.333 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:50:43.333 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T08:50:43.333 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T08:50:43.333 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T08:50:43.333 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:43.335 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T08:50:43.418 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T08:50:43.418 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T08:50:43.965 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T08:50:43.965 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T08:50:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.967 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:43.968 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:50:44.377 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test 2024-09-16T08:50:44.541 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T08:50:44.541 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 1/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 2/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.3.0-4917.g8293d73f.el9.x86_64 3/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.3.0-4917.g8293d73f.el9.x86_64 4/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 5/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.3.0-4917.g8293d73f.el9.x86_6 6/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.3.0-4917.g8293d73f.el9.x86_64 7/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4917.g8293d73f.el9. 8/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.3.0-4917.g8293d73f.el9.x8 9/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.3.0-4917.g8293d73f.el9.x86_64 10/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.3.0-4917.g8293d73f.el9.x86_64 11/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.3.0-4917.g8293d73f.el9.x86_64 12/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.3.0-4917.g8293d73f.el9.x86_64 13/35 2024-09-16T08:50:44.542 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T08:50:44.543 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T08:50:44.544 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T08:50:44.544 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T08:50:44.545 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T08:50:44.545 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T08:50:44.545 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 32/35 2024-09-16T08:50:44.545 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T08:50:44.545 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 34/35 2024-09-16T08:50:44.946 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:50:45.078 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:50:45.078 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:50:45.078 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 50 M 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 161 k 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout:Install 4 Packages 2024-09-16T08:50:45.079 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:45.080 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 51 M 2024-09-16T08:50:45.080 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 217 M 2024-09-16T08:50:45.080 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:50:45.437 INFO:teuthology.orchestra.run.smithi111.stdout:(1/4): libcephsqlite-19.3.0-4917.g8293d73f.el9. 684 kB/s | 161 kB 00:00 2024-09-16T08:50:45.495 INFO:teuthology.orchestra.run.smithi111.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.0 MB/s | 304 kB 00:00 2024-09-16T08:50:45.589 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T08:50:45.589 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:45.589 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-09-16T08:50:45.589 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.589 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.589 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:50:45.589 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T08:50:45.589 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.589 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T08:50:45.590 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T08:50:45.591 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T08:50:45.592 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T08:50:45.592 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:45.592 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:50:45.595 INFO:teuthology.orchestra.run.smithi111.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 402 kB/s | 64 kB 00:00 2024-09-16T08:50:45.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-09-16T08:50:46.491 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:30 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:50:46.621 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:50:46.622 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:50:46.622 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:50:46.622 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:50:46.622 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:50:46.622 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 50 M 2024-09-16T08:50:46.622 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:50:46.622 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 161 k 2024-09-16T08:50:46.622 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T08:50:46.622 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T08:50:46.623 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:46.623 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:50:46.623 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:50:46.623 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-16T08:50:46.623 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:46.623 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 51 M 2024-09-16T08:50:46.623 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 217 M 2024-09-16T08:50:46.624 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:50:46.820 INFO:teuthology.orchestra.run.smithi111.stdout:(4/4): ceph-test-19.3.0-4917.g8293d73f.el9.x86_ 31 MB/s | 50 MB 00:01 2024-09-16T08:50:46.822 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:50:46.823 INFO:teuthology.orchestra.run.smithi111.stdout:Total 29 MB/s | 51 MB 00:01 2024-09-16T08:50:46.851 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:50:46.871 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:50:46.871 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:50:47.122 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): libcephsqlite-19.3.0-4917.g8293d73f.el9. 730 kB/s | 161 kB 00:00 2024-09-16T08:50:47.277 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:50:47.277 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:50:47.681 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 390 kB/s | 304 kB 00:00 2024-09-16T08:50:47.756 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 102 kB/s | 64 kB 00:00 2024-09-16T08:50:47.864 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:50:47.950 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T08:50:48.000 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T08:50:48.030 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:19.3.0-4917.g8293d73f.el9.x86_64 3/4 2024-09-16T08:50:48.481 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): ceph-test-19.3.0-4917.g8293d73f.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-16T08:50:48.483 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:50:48.484 INFO:teuthology.orchestra.run.smithi114.stdout:Total 27 MB/s | 51 MB 00:01 2024-09-16T08:50:48.511 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:50:48.532 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:50:48.532 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:50:48.945 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:50:48.946 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:50:49.551 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:50:49.641 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T08:50:49.708 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T08:50:49.733 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.3.0-4917.g8293d73f.el9.x86_64 3/4 2024-09-16T08:50:52.787 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4917.g8293d73f.el9.x86_64 3/4 2024-09-16T08:50:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:19.3.0-4917.g8293d73f.el9.x86_64 4/4 2024-09-16T08:50:54.470 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4917.g8293d73f.el9.x86_64 3/4 2024-09-16T08:50:54.558 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.3.0-4917.g8293d73f.el9.x86_64 4/4 2024-09-16T08:50:55.363 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-2:19.3.0-4917.g8293d73f.el9.x86_64 4/4 2024-09-16T08:50:55.363 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:19.3.0-4917.g8293d73f.el9.x86_64 1/4 2024-09-16T08:50:55.363 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:19.3.0-4917.g8293d73f.el9.x86_64 2/4 2024-09-16T08:50:55.363 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T08:50:55.667 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T08:50:55.667 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:55.667 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:50:55.667 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:55.667 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:55.667 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T08:50:55.667 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T08:50:55.667 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:55.668 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:50:55.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph 2024-09-16T08:50:56.565 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:50:56.707 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:50:56.710 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:50:56.710 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:50:56.710 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:50:56.710 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:50:56.710 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 6.6 k 2024-09-16T08:50:56.710 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:50:56.710 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 2.4 M 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 1.5 M 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 291 k 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 4.8 M 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 17 M 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T08:50:56.711 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T08:50:56.712 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:56.713 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:50:56.714 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:50:56.714 INFO:teuthology.orchestra.run.smithi111.stdout:Install 37 Packages 2024-09-16T08:50:56.714 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:50:56.714 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 31 M 2024-09-16T08:50:56.714 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 109 M 2024-09-16T08:50:56.714 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:50:57.086 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:19.3.0-4917.g8293d73f.el9.x86_64 4/4 2024-09-16T08:50:57.086 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.3.0-4917.g8293d73f.el9.x86_64 1/4 2024-09-16T08:50:57.086 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.3.0-4917.g8293d73f.el9.x86_64 2/4 2024-09-16T08:50:57.086 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T08:50:57.393 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T08:50:57.393 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:57.393 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:50:57.393 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:57.393 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:50:57.393 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T08:50:57.394 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T08:50:57.394 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:57.394 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:50:57.564 INFO:teuthology.orchestra.run.smithi111.stdout:(1/37): ceph-19.3.0-4917.g8293d73f.el9.x86_64.r 51 kB/s | 6.6 kB 00:00 2024-09-16T08:50:57.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-09-16T08:50:57.814 INFO:teuthology.orchestra.run.smithi111.stdout:(2/37): ceph-mds-19.3.0-4917.g8293d73f.el9.x86_ 6.3 MB/s | 2.4 MB 00:00 2024-09-16T08:50:57.856 INFO:teuthology.orchestra.run.smithi111.stdout:(3/37): ceph-mgr-19.3.0-4917.g8293d73f.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2024-09-16T08:50:57.948 INFO:teuthology.orchestra.run.smithi111.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4917.g8293 3.1 MB/s | 291 kB 00:00 2024-09-16T08:50:58.023 INFO:teuthology.orchestra.run.smithi111.stdout:(5/37): ceph-mon-19.3.0-4917.g8293d73f.el9.x86_ 10 MB/s | 4.8 MB 00:00 2024-09-16T08:50:58.174 INFO:teuthology.orchestra.run.smithi111.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-16T08:50:58.236 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:42 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:50:58.366 INFO:teuthology.orchestra.run.smithi111.stdout:(7/37): ceph-osd-19.3.0-4917.g8293d73f.el9.x86_ 31 MB/s | 17 MB 00:00 2024-09-16T08:50:58.375 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:50:58.377 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:50:58.377 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:50:58.377 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 6.6 k 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 2.4 M 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 1.5 M 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 291 k 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 4.8 M 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 17 M 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T08:50:58.378 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T08:50:58.379 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T08:50:58.380 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T08:50:58.381 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:58.381 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:50:58.381 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:50:58.381 INFO:teuthology.orchestra.run.smithi114.stdout:Install 37 Packages 2024-09-16T08:50:58.381 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:50:58.381 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 31 M 2024-09-16T08:50:58.382 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 109 M 2024-09-16T08:50:58.382 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:50:58.390 INFO:teuthology.orchestra.run.smithi111.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-09-16T08:50:58.432 INFO:teuthology.orchestra.run.smithi111.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 3.0 MB/s | 1.2 MB 00:00 2024-09-16T08:50:58.458 INFO:teuthology.orchestra.run.smithi111.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-16T08:50:58.483 INFO:teuthology.orchestra.run.smithi111.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-16T08:50:58.517 INFO:teuthology.orchestra.run.smithi111.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-09-16T08:50:58.542 INFO:teuthology.orchestra.run.smithi111.stdout:(13/37): python3-pycparser-2.20-6.el9.noarch.rp 767 kB/s | 135 kB 00:00 2024-09-16T08:50:58.559 INFO:teuthology.orchestra.run.smithi111.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 833 kB/s | 35 kB 00:00 2024-09-16T08:50:58.584 INFO:teuthology.orchestra.run.smithi111.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-16T08:50:58.601 INFO:teuthology.orchestra.run.smithi111.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-16T08:50:58.626 INFO:teuthology.orchestra.run.smithi111.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 920 kB/s | 77 kB 00:00 2024-09-16T08:50:58.652 INFO:teuthology.orchestra.run.smithi111.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 618 kB/s | 42 kB 00:00 2024-09-16T08:50:58.694 INFO:teuthology.orchestra.run.smithi111.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-16T08:50:58.719 INFO:teuthology.orchestra.run.smithi111.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 649 kB/s | 43 kB 00:00 2024-09-16T08:50:58.744 INFO:teuthology.orchestra.run.smithi111.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.5 MB/s | 172 kB 00:00 2024-09-16T08:50:58.770 INFO:teuthology.orchestra.run.smithi111.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 132 kB/s | 22 kB 00:00 2024-09-16T08:50:58.795 INFO:teuthology.orchestra.run.smithi111.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-16T08:50:58.828 INFO:teuthology.orchestra.run.smithi111.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 876 kB/s | 23 kB 00:00 2024-09-16T08:50:58.853 INFO:teuthology.orchestra.run.smithi111.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 841 kB/s | 19 kB 00:00 2024-09-16T08:50:58.879 INFO:teuthology.orchestra.run.smithi111.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-09-16T08:50:58.904 INFO:teuthology.orchestra.run.smithi111.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-16T08:50:58.929 INFO:teuthology.orchestra.run.smithi111.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-09-16T08:50:58.955 INFO:teuthology.orchestra.run.smithi111.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-16T08:50:58.980 INFO:teuthology.orchestra.run.smithi111.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-16T08:50:59.005 INFO:teuthology.orchestra.run.smithi111.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-16T08:50:59.030 INFO:teuthology.orchestra.run.smithi111.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-16T08:50:59.056 INFO:teuthology.orchestra.run.smithi111.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-16T08:50:59.081 INFO:teuthology.orchestra.run.smithi111.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-16T08:50:59.106 INFO:teuthology.orchestra.run.smithi111.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-16T08:50:59.131 INFO:teuthology.orchestra.run.smithi111.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T08:50:59.150 INFO:teuthology.orchestra.run.smithi114.stdout:(1/37): ceph-19.3.0-4917.g8293d73f.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-16T08:50:59.165 INFO:teuthology.orchestra.run.smithi111.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T08:50:59.169 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:50:59.169 INFO:teuthology.orchestra.run.smithi111.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-16T08:50:59.300 INFO:teuthology.orchestra.run.smithi114.stdout:(2/37): ceph-mgr-19.3.0-4917.g8293d73f.el9.x86_ 5.4 MB/s | 1.5 MB 00:00 2024-09-16T08:50:59.318 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:50:59.359 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:50:59.359 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:50:59.375 INFO:teuthology.orchestra.run.smithi114.stdout:(3/37): ceph-mds-19.3.0-4917.g8293d73f.el9.x86_ 6.6 MB/s | 2.4 MB 00:00 2024-09-16T08:50:59.434 INFO:teuthology.orchestra.run.smithi114.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4917.g8293 4.9 MB/s | 291 kB 00:00 2024-09-16T08:50:59.676 INFO:teuthology.orchestra.run.smithi114.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-16T08:50:59.816 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:50:59.817 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:50:59.859 INFO:teuthology.orchestra.run.smithi114.stdout:(6/37): ceph-osd-19.3.0-4917.g8293d73f.el9.x86_ 31 MB/s | 17 MB 00:00 2024-09-16T08:50:59.935 INFO:teuthology.orchestra.run.smithi114.stdout:(7/37): ceph-mon-19.3.0-4917.g8293d73f.el9.x86_ 6.1 MB/s | 4.8 MB 00:00 2024-09-16T08:51:00.060 INFO:teuthology.orchestra.run.smithi114.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 3.2 MB/s | 1.2 MB 00:00 2024-09-16T08:51:00.085 INFO:teuthology.orchestra.run.smithi114.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 472 kB/s | 106 kB 00:00 2024-09-16T08:51:00.110 INFO:teuthology.orchestra.run.smithi114.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-16T08:51:00.144 INFO:teuthology.orchestra.run.smithi114.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 646 kB/s | 135 kB 00:00 2024-09-16T08:51:00.169 INFO:teuthology.orchestra.run.smithi114.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-16T08:51:00.270 INFO:teuthology.orchestra.run.smithi114.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-16T08:51:00.295 INFO:teuthology.orchestra.run.smithi114.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 277 kB/s | 35 kB 00:00 2024-09-16T08:51:00.329 INFO:teuthology.orchestra.run.smithi114.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 934 kB/s | 172 kB 00:00 2024-09-16T08:51:00.354 INFO:teuthology.orchestra.run.smithi114.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 922 kB/s | 77 kB 00:00 2024-09-16T08:51:00.379 INFO:teuthology.orchestra.run.smithi114.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 607 kB/s | 51 kB 00:00 2024-09-16T08:51:00.404 INFO:teuthology.orchestra.run.smithi114.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-16T08:51:00.446 INFO:teuthology.orchestra.run.smithi114.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-16T08:51:00.472 INFO:teuthology.orchestra.run.smithi114.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.2 MB/s | 172 kB 00:00 2024-09-16T08:51:00.497 INFO:teuthology.orchestra.run.smithi114.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-09-16T08:51:00.522 INFO:teuthology.orchestra.run.smithi114.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-16T08:51:00.547 INFO:teuthology.orchestra.run.smithi114.stdout:(23/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.6 MB/s | 358 kB 00:00 2024-09-16T08:51:00.573 INFO:teuthology.orchestra.run.smithi114.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-16T08:51:00.590 INFO:teuthology.orchestra.run.smithi114.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 552 kB/s | 23 kB 00:00 2024-09-16T08:51:00.606 INFO:teuthology.orchestra.run.smithi114.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 581 kB/s | 19 kB 00:00 2024-09-16T08:51:00.623 INFO:teuthology.orchestra.run.smithi114.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 589 kB/s | 20 kB 00:00 2024-09-16T08:51:00.640 INFO:teuthology.orchestra.run.smithi114.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-16T08:51:00.666 INFO:teuthology.orchestra.run.smithi114.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-16T08:51:00.682 INFO:teuthology.orchestra.run.smithi114.stdout:(30/37): lua-devel-5.4.4-4.el9.x86_64.rpm 68 kB/s | 22 kB 00:00 2024-09-16T08:51:00.708 INFO:teuthology.orchestra.run.smithi114.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-09-16T08:51:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.4 MB/s | 36 kB 00:00 2024-09-16T08:51:00.750 INFO:teuthology.orchestra.run.smithi114.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-16T08:51:00.775 INFO:teuthology.orchestra.run.smithi114.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-16T08:51:00.801 INFO:teuthology.orchestra.run.smithi114.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-16T08:51:00.826 INFO:teuthology.orchestra.run.smithi114.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-09-16T08:51:00.851 INFO:teuthology.orchestra.run.smithi114.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-16T08:51:00.854 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:00.854 INFO:teuthology.orchestra.run.smithi114.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-16T08:51:01.003 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:01.043 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:01.044 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:01.131 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:01.184 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T08:51:01.217 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T08:51:01.245 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T08:51:01.317 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T08:51:01.361 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T08:51:01.402 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T08:51:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T08:51:01.495 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:01.495 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:01.573 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T08:51:01.613 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T08:51:01.656 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T08:51:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T08:51:01.721 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T08:51:01.754 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T08:51:01.794 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T08:51:01.829 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T08:51:01.864 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T08:51:01.912 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T08:51:01.944 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T08:51:02.018 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T08:51:02.165 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T08:51:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T08:51:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T08:51:02.344 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T08:51:02.409 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T08:51:02.464 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T08:51:02.652 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T08:51:02.698 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T08:51:02.776 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T08:51:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T08:51:02.878 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T08:51:02.911 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T08:51:02.911 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T08:51:02.939 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T08:51:03.012 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T08:51:03.042 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T08:51:03.072 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T08:51:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T08:51:03.196 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T08:51:03.203 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4917.g8293d73f.el9. 32/37 2024-09-16T08:51:03.232 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 33/37 2024-09-16T08:51:03.259 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 33/37 2024-09-16T08:51:03.259 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:03.259 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T08:51:03.259 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T08:51:03.259 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T08:51:03.259 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:03.275 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T08:51:03.332 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T08:51:03.367 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T08:51:03.397 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T08:51:03.432 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T08:51:03.465 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T08:51:03.497 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T08:51:03.532 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T08:51:03.566 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T08:51:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T08:51:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T08:51:03.721 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T08:51:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T08:51:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T08:51:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T08:51:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T08:51:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T08:51:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T08:51:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T08:51:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T08:51:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T08:51:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T08:51:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T08:51:04.725 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:19.3.0-4917.g8293d73f.el9.x86_64 34/37 2024-09-16T08:51:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T08:51:04.752 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:19.3.0-4917.g8293d73f.el9.x86_64 34/37 2024-09-16T08:51:04.752 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:04.752 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T08:51:04.752 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T08:51:04.753 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T08:51:04.753 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4917.g8293d73f.el9. 32/37 2024-09-16T08:51:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 33/37 2024-09-16T08:51:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 33/37 2024-09-16T08:51:04.979 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:04.979 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T08:51:04.979 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T08:51:04.979 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T08:51:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:05.200 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:19.3.0-4917.g8293d73f.el9.x86_64 35/37 2024-09-16T08:51:05.225 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:19.3.0-4917.g8293d73f.el9.x86_64 35/37 2024-09-16T08:51:05.225 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:05.225 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T08:51:05.225 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T08:51:05.225 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T08:51:05.225 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:19.3.0-4917.g8293d73f.el9.x86_64 36/37 2024-09-16T08:51:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:19.3.0-4917.g8293d73f.el9.x86_64 36/37 2024-09-16T08:51:05.482 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:05.482 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T08:51:05.483 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T08:51:05.483 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T08:51:05.483 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:05.540 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:19.3.0-4917.g8293d73f.el9.x86_64 37/37 2024-09-16T08:51:06.444 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.3.0-4917.g8293d73f.el9.x86_64 34/37 2024-09-16T08:51:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.3.0-4917.g8293d73f.el9.x86_64 34/37 2024-09-16T08:51:06.472 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:06.472 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T08:51:06.472 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T08:51:06.472 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T08:51:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:06.920 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.3.0-4917.g8293d73f.el9.x86_64 35/37 2024-09-16T08:51:06.944 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.3.0-4917.g8293d73f.el9.x86_64 35/37 2024-09-16T08:51:06.944 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:06.944 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T08:51:06.944 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T08:51:06.944 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T08:51:06.944 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:07.171 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.3.0-4917.g8293d73f.el9.x86_64 36/37 2024-09-16T08:51:07.195 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.3.0-4917.g8293d73f.el9.x86_64 36/37 2024-09-16T08:51:07.195 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:07.195 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T08:51:07.195 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T08:51:07.195 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T08:51:07.195 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:07.229 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-2:19.3.0-4917.g8293d73f.el9.x86_64 37/37 2024-09-16T08:51:07.229 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:19.3.0-4917.g8293d73f.el9.x86_64 1/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:19.3.0-4917.g8293d73f.el9.x86_64 2/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 3/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:19.3.0-4917.g8293d73f.el9.x86_64 4/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:19.3.0-4917.g8293d73f.el9.x86_64 5/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4917.g8293d73f.el9. 6/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T08:51:07.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T08:51:07.231 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T08:51:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T08:51:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T08:51:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T08:51:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T08:51:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T08:51:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T08:51:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T08:51:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T08:51:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T08:51:07.234 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T08:51:07.234 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T08:51:07.268 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.3.0-4917.g8293d73f.el9.x86_64 37/37 2024-09-16T08:51:08.245 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T08:51:08.245 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:08.245 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:08.245 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:08.245 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:08.245 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:08.245 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:08.245 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:08.245 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T08:51:08.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T08:51:08.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T08:51:08.248 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T08:51:08.248 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T08:51:08.248 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T08:51:08.248 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T08:51:08.248 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:08.248 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:08.650 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base 2024-09-16T08:51:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:19.3.0-4917.g8293d73f.el9.x86_64 37/37 2024-09-16T08:51:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.3.0-4917.g8293d73f.el9.x86_64 1/37 2024-09-16T08:51:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.3.0-4917.g8293d73f.el9.x86_64 2/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 3/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.3.0-4917.g8293d73f.el9.x86_64 4/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.3.0-4917.g8293d73f.el9.x86_64 5/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4917.g8293d73f.el9. 6/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T08:51:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T08:51:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T08:51:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T08:51:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T08:51:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T08:51:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T08:51:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T08:51:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T08:51:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T08:51:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T08:51:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T08:51:09.216 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:09.307 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:09.349 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:09.350 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-09-16T08:51:09.350 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:09.429 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install cephadm 2024-09-16T08:51:09.989 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:10.122 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:10.122 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 784 k 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 784 k 2024-09-16T08:51:10.123 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 791 k 2024-09-16T08:51:10.124 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:10.239 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T08:51:10.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:10.239 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:10.239 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T08:51:10.240 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T08:51:10.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:10.242 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:10.408 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm-19.3.0-4917.g8293d73f.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-16T08:51:10.408 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:10.409 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-16T08:51:10.409 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:10.414 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:10.414 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:10.420 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:10.420 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:10.497 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:10.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-09-16T08:51:10.914 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:11.005 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:11.239 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:11.328 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:11.370 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:11.371 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T08:51:11.371 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:11.387 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:11.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-09-16T08:51:11.640 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:11.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:11.640 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:11.640 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:11.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:11.640 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:11.855 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T08:51:12.003 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:12.135 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:12.136 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:12.136 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:51:12.136 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:12.136 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:12.136 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 784 k 2024-09-16T08:51:12.136 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:12.136 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:12.137 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:12.137 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T08:51:12.137 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:12.137 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 784 k 2024-09-16T08:51:12.137 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 791 k 2024-09-16T08:51:12.137 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:12.412 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-19.3.0-4917.g8293d73f.el9.noarch.rpm 2.8 MB/s | 784 kB 00:00 2024-09-16T08:51:12.413 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:12.413 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.8 MB/s | 784 kB 00:00 2024-09-16T08:51:12.413 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:12.418 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:12.418 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:12.425 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:12.425 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:12.434 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:12.521 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:12.572 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 145 k 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-09-16T08:51:12.573 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:12.574 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 145 k 2024-09-16T08:51:12.574 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 432 k 2024-09-16T08:51:12.574 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:12.782 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-19.3.0-4917.g8293d7 697 kB/s | 145 kB 00:00 2024-09-16T08:51:12.783 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:12.783 INFO:teuthology.orchestra.run.smithi111.stdout:Total 692 kB/s | 145 kB 00:00 2024-09-16T08:51:12.783 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:12.791 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:12.791 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:12.850 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:12.850 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:12.974 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:12.996 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:13.050 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4917.g8293d73f. 1/1 2024-09-16T08:51:13.076 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4917.g8293d73f. 1/1 2024-09-16T08:51:13.076 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:13.076 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T08:51:13.076 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:13.091 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:13.452 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:13.698 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:13.699 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:13.699 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:13.699 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:13.699 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:13.699 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:13.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T08:51:13.998 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4917.g8293d73f. 1/1 2024-09-16T08:51:13.998 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:13.998 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:13.998 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:13.999 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:13.999 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:14.245 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr 2024-09-16T08:51:14.484 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:14.619 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 145 k 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 145 k 2024-09-16T08:51:14.620 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 432 k 2024-09-16T08:51:14.621 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:14.824 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:14.828 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-19.3.0-4917.g8293d7 699 kB/s | 145 kB 00:00 2024-09-16T08:51:14.829 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:14.829 INFO:teuthology.orchestra.run.smithi114.stdout:Total 693 kB/s | 145 kB 00:00 2024-09-16T08:51:14.830 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:14.837 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:14.837 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:14.894 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:14.894 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:14.915 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:14.957 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:14.958 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-09-16T08:51:14.958 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:15.035 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T08:51:15.052 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:15.138 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4917.g8293d73f. 1/1 2024-09-16T08:51:15.163 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4917.g8293d73f. 1/1 2024-09-16T08:51:15.163 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:15.163 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T08:51:15.163 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:15.599 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:15.733 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:15.734 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:51:15.734 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:15.734 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 14 M 2024-09-16T08:51:15.734 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 34 k 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 16 k 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:15.735 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:15.736 INFO:teuthology.orchestra.run.smithi111.stdout:Install 12 Packages 2024-09-16T08:51:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:15.736 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 19 M 2024-09-16T08:51:15.736 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 159 M 2024-09-16T08:51:15.736 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:16.098 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4917.g8293d73f. 1/1 2024-09-16T08:51:16.098 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:16.098 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:16.099 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:16.099 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:16.099 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:16.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-09-16T08:51:16.361 INFO:teuthology.orchestra.run.smithi111.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4917.g82 227 kB/s | 34 kB 00:00 2024-09-16T08:51:16.378 INFO:teuthology.orchestra.run.smithi111.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4917.g829 98 kB/s | 16 kB 00:00 2024-09-16T08:51:16.562 INFO:teuthology.orchestra.run.smithi111.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-16T08:51:16.612 INFO:teuthology.orchestra.run.smithi111.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-09-16T08:51:16.671 INFO:teuthology.orchestra.run.smithi111.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T08:51:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-09-16T08:51:16.855 INFO:teuthology.orchestra.run.smithi111.stdout:(7/12): ceph-mgr-dashboard-19.3.0-4917.g8293d73 22 MB/s | 14 MB 00:00 2024-09-16T08:51:16.888 INFO:teuthology.orchestra.run.smithi111.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-09-16T08:51:16.902 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:16.955 INFO:teuthology.orchestra.run.smithi111.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.9 MB/s | 2.0 MB 00:00 2024-09-16T08:51:16.989 INFO:teuthology.orchestra.run.smithi111.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-09-16T08:51:16.997 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:17.014 INFO:teuthology.orchestra.run.smithi111.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.6 MB/s | 188 kB 00:00 2024-09-16T08:51:17.030 INFO:teuthology.orchestra.run.smithi111.stdout:(12/12): python3-repoze-lru-0.7-16.el9.noarch.r 219 kB/s | 31 kB 00:00 2024-09-16T08:51:17.034 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:17.034 INFO:teuthology.orchestra.run.smithi111.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-16T08:51:17.040 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:17.041 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T08:51:17.041 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:17.083 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:17.101 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:17.101 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:17.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T08:51:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:17.664 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:17.796 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:17.797 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:17.797 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:51:17.797 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:17.797 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:17.797 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 14 M 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 34 k 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 16 k 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T08:51:17.798 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T08:51:17.799 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:17.799 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:17.799 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:17.799 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-09-16T08:51:17.799 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:17.799 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 19 M 2024-09-16T08:51:17.799 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 159 M 2024-09-16T08:51:17.800 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:17.964 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:18.031 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T08:51:18.066 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T08:51:18.113 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T08:51:18.143 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T08:51:18.190 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4917.g82 243 kB/s | 34 kB 00:00 2024-09-16T08:51:18.215 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4917.g829 99 kB/s | 16 kB 00:00 2024-09-16T08:51:18.240 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T08:51:18.366 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 269 kB 00:00 2024-09-16T08:51:18.369 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T08:51:18.408 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-09-16T08:51:18.458 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-16T08:51:18.471 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T08:51:18.483 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-16T08:51:18.517 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.6 MB/s | 863 kB 00:00 2024-09-16T08:51:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T08:51:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T08:51:18.567 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-09-16T08:51:18.592 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4917.g8293d73f.el9 10/12 2024-09-16T08:51:18.592 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T08:51:18.618 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-16T08:51:18.676 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-16T08:51:18.901 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4917.g8293d7 17 MB/s | 14 MB 00:00 2024-09-16T08:51:18.905 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:18.905 INFO:teuthology.orchestra.run.smithi114.stdout:Total 17 MB/s | 19 MB 00:01 2024-09-16T08:51:18.953 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:18.971 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:18.971 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:19.250 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:19.250 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:19.832 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:19.910 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T08:51:19.945 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T08:51:19.993 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T08:51:20.022 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T08:51:20.122 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T08:51:20.258 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T08:51:20.351 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T08:51:20.411 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T08:51:20.439 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T08:51:20.471 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4917.g8293d73f.el9 10/12 2024-09-16T08:51:21.763 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4917.g8293d73f.el 11/12 2024-09-16T08:51:21.805 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4917.g8293d73f.el9.noa 12/12 2024-09-16T08:51:23.668 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4917.g8293d73f.el 11/12 2024-09-16T08:51:23.718 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4917.g8293d73f.el9.noa 12/12 2024-09-16T08:51:23.796 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4917.g8293d73f.el9.noa 12/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4917.g8293d73f.el 1/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4917.g8293d73f.el9.noa 2/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4917.g8293d73f.el9 3/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T08:51:23.797 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T08:51:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T08:51:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:24.121 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T08:51:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:24.122 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:24.389 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T08:51:24.954 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:08 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:25.088 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 7.4 M 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T08:51:25.089 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout:Install 12 Packages 2024-09-16T08:51:25.090 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:25.091 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 38 M 2024-09-16T08:51:25.091 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 212 M 2024-09-16T08:51:25.091 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:25.395 INFO:teuthology.orchestra.run.smithi111.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-16T08:51:25.437 INFO:teuthology.orchestra.run.smithi111.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-16T08:51:25.520 INFO:teuthology.orchestra.run.smithi111.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-16T08:51:25.571 INFO:teuthology.orchestra.run.smithi111.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-16T08:51:25.621 INFO:teuthology.orchestra.run.smithi111.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 767 kB/s | 38 kB 00:00 2024-09-16T08:51:25.713 INFO:teuthology.orchestra.run.smithi111.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-16T08:51:25.789 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4917.g8293d73f.el9.noa 12/12 2024-09-16T08:51:25.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4917.g8293d73f.el 1/12 2024-09-16T08:51:25.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4917.g8293d73f.el9.noa 2/12 2024-09-16T08:51:25.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4917.g8293d73f.el9 3/12 2024-09-16T08:51:25.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T08:51:25.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T08:51:25.790 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T08:51:25.790 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T08:51:25.790 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T08:51:25.790 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T08:51:25.790 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T08:51:25.790 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T08:51:25.880 INFO:teuthology.orchestra.run.smithi111.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.8 MB/s | 3.0 MB 00:00 2024-09-16T08:51:25.980 INFO:teuthology.orchestra.run.smithi111.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 917 kB/s | 245 kB 00:00 2024-09-16T08:51:26.047 INFO:teuthology.orchestra.run.smithi111.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2024-09-16T08:51:26.098 INFO:teuthology.orchestra.run.smithi111.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.1 MB/s | 250 kB 00:00 2024-09-16T08:51:26.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T08:51:26.115 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T08:51:26.116 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:26.117 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:26.223 INFO:teuthology.orchestra.run.smithi111.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-09-16T08:51:26.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T08:51:26.673 INFO:teuthology.orchestra.run.smithi111.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-09-16T08:51:26.674 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:26.674 INFO:teuthology.orchestra.run.smithi111.stdout:Total 24 MB/s | 38 MB 00:01 2024-09-16T08:51:26.945 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:10 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:27.030 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:27.057 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:27.057 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:27.078 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 7.4 M 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T08:51:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T08:51:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T08:51:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T08:51:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T08:51:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T08:51:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:27.081 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:27.081 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-09-16T08:51:27.081 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-09-16T08:51:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:27.082 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 38 M 2024-09-16T08:51:27.082 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 212 M 2024-09-16T08:51:27.082 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:27.324 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-16T08:51:27.358 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-16T08:51:27.458 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-16T08:51:27.491 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:27.491 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:27.509 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-16T08:51:27.559 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 773 kB/s | 38 kB 00:00 2024-09-16T08:51:27.668 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-49 15 MB/s | 7.4 MB 00:00 2024-09-16T08:51:27.868 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-16T08:51:28.258 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:28.327 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T08:51:28.371 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T08:51:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T08:51:28.514 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T08:51:28.576 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-09-16T08:51:28.698 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T08:51:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 250 kB 00:00 2024-09-16T08:51:28.737 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T08:51:28.824 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T08:51:29.194 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.0 MB/s | 4.9 MB 00:01 2024-09-16T08:51:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T08:51:29.592 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T08:51:29.985 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.4 MB/s | 5.1 MB 00:02 2024-09-16T08:51:30.852 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T08:51:31.344 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.2 MB/s | 16 MB 00:02 2024-09-16T08:51:31.344 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:31.345 INFO:teuthology.orchestra.run.smithi114.stdout:Total 9.0 MB/s | 38 MB 00:04 2024-09-16T08:51:31.702 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:31.728 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:31.728 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:31.889 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T08:51:31.920 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4917.g8293d 12/12 2024-09-16T08:51:32.156 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:32.157 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:32.915 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:32.978 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T08:51:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T08:51:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T08:51:33.176 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T08:51:33.363 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T08:51:33.392 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T08:51:33.469 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T08:51:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T08:51:34.222 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T08:51:34.537 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4917.g8293d 12/12 2024-09-16T08:51:34.537 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4917.g8293d 1/12 2024-09-16T08:51:34.537 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T08:51:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T08:51:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T08:51:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T08:51:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T08:51:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T08:51:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T08:51:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T08:51:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T08:51:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T08:51:34.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T08:51:34.939 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:34.939 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:34.939 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:34.939 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:35.306 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-rook 2024-09-16T08:51:35.456 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T08:51:35.887 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:19 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:36.022 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:36.023 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:36.023 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:51:36.023 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:36.023 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:36.023 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 49 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T08:51:36.024 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T08:51:36.025 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:36.025 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:36.025 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:36.025 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-09-16T08:51:36.025 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:36.025 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 2.2 M 2024-09-16T08:51:36.026 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 26 M 2024-09-16T08:51:36.026 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:36.187 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 233 kB/s | 19 kB 00:00 2024-09-16T08:51:36.204 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 259 kB/s | 26 kB 00:00 2024-09-16T08:51:36.263 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-16T08:51:36.288 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): ceph-mgr-rook-19.3.0-4917.g8293d73f.el9 269 kB/s | 49 kB 00:00 2024-09-16T08:51:36.313 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-16T08:51:36.338 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-09-16T08:51:36.380 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-16T08:51:36.405 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 457 kB/s | 54 kB 00:00 2024-09-16T08:51:36.430 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 165 kB/s | 14 kB 00:00 2024-09-16T08:51:36.464 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-16T08:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T08:51:36.490 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-16T08:51:36.515 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T08:51:36.516 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4917.g8293d 12/12 2024-09-16T08:51:36.598 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2024-09-16T08:51:36.601 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:36.601 INFO:teuthology.orchestra.run.smithi111.stdout:Total 3.8 MB/s | 2.2 MB 00:00 2024-09-16T08:51:36.664 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:36.680 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:36.680 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:36.766 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:36.767 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:37.274 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:37.473 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T08:51:37.525 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T08:51:37.580 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T08:51:37.617 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T08:51:37.659 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T08:51:37.761 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T08:51:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T08:51:37.936 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T08:51:38.661 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T08:51:38.707 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T08:51:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T08:51:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T08:51:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:19.3.0-4917.g8293d73f.el9.noarch 13/13 2024-09-16T08:51:39.139 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4917.g8293d 12/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4917.g8293d 1/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T08:51:39.140 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T08:51:39.141 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T08:51:39.467 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4917.g8293d73f.el9.noarch 13/13 2024-09-16T08:51:39.467 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4917.g8293d73f.el9.noarch 1/13 2024-09-16T08:51:39.467 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T08:51:39.467 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T08:51:39.467 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T08:51:39.467 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T08:51:39.467 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T08:51:39.468 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T08:51:39.468 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T08:51:39.468 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T08:51:39.468 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T08:51:39.468 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T08:51:39.468 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T08:51:39.624 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T08:51:39.625 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T08:51:39.626 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T08:51:39.626 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:39.626 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:39.804 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T08:51:39.804 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:39.804 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:39.804 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:39.804 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T08:51:39.804 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T08:51:39.804 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T08:51:39.804 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:39.805 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:39.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-09-16T08:51:40.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T08:51:40.586 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:24 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:40.622 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:24 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:40.722 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:40.723 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:40.723 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:51:40.723 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:40.723 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:40.723 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 49 k 2024-09-16T08:51:40.723 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:51:40.723 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T08:51:40.723 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:40.724 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:40.725 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:40.725 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-09-16T08:51:40.725 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:40.725 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.2 M 2024-09-16T08:51:40.725 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-09-16T08:51:40.725 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:40.757 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 154 k 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T08:51:40.758 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T08:51:40.759 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T08:51:40.759 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:40.759 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:40.759 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:40.759 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-09-16T08:51:40.759 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:40.759 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 7.0 M 2024-09-16T08:51:40.759 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 33 M 2024-09-16T08:51:40.760 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:40.909 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 182 kB/s | 19 kB 00:00 2024-09-16T08:51:40.934 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 196 kB/s | 26 kB 00:00 2024-09-16T08:51:40.993 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): ceph-mgr-rook-19.3.0-4917.g8293d73f.el9 260 kB/s | 49 kB 00:00 2024-09-16T08:51:41.009 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-16T08:51:41.035 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-09-16T08:51:41.068 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 963 kB/s | 32 kB 00:00 2024-09-16T08:51:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 708 kB/s | 54 kB 00:00 2024-09-16T08:51:41.102 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 419 kB/s | 14 kB 00:00 2024-09-16T08:51:41.144 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-09-16T08:51:41.178 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.3 MB/s | 220 kB 00:00 2024-09-16T08:51:41.220 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 9.2 MB/s | 1.0 MB 00:00 2024-09-16T08:51:41.245 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-09-16T08:51:41.270 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 993 kB/s | 90 kB 00:00 2024-09-16T08:51:41.272 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:41.273 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.0 MB/s | 2.2 MB 00:00 2024-09-16T08:51:41.335 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:41.342 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T08:51:41.350 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:41.350 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:41.367 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4917.g8293d73f.e 688 kB/s | 154 kB 00:00 2024-09-16T08:51:41.417 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-16T08:51:41.437 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:41.439 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:41.442 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 767 kB/s | 58 kB 00:00 2024-09-16T08:51:41.468 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-16T08:51:41.626 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-09-16T08:51:41.629 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:41.629 INFO:teuthology.orchestra.run.smithi111.stdout:Total 8.1 MB/s | 7.0 MB 00:00 2024-09-16T08:51:41.700 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:41.712 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:41.712 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:41.813 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:41.814 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:41.957 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:42.132 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T08:51:42.136 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:42.188 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T08:51:42.244 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T08:51:42.281 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T08:51:42.299 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T08:51:42.322 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T08:51:42.345 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T08:51:42.422 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T08:51:42.533 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T08:51:42.581 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T08:51:42.869 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T08:51:42.956 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T08:51:43.024 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T08:51:43.049 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 6/6 2024-09-16T08:51:43.297 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T08:51:43.346 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T08:51:43.379 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T08:51:43.427 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T08:51:43.449 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.3.0-4917.g8293d73f.el9.noarch 13/13 2024-09-16T08:51:43.836 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 6/6 2024-09-16T08:51:43.836 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/6 2024-09-16T08:51:43.836 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T08:51:43.836 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T08:51:43.836 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T08:51:43.836 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T08:51:44.110 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:44.111 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4917.g8293d73f.el9.noarch 13/13 2024-09-16T08:51:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4917.g8293d73f.el9.noarch 1/13 2024-09-16T08:51:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T08:51:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T08:51:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T08:51:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T08:51:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T08:51:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T08:51:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T08:51:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T08:51:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T08:51:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T08:51:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T08:51:44.372 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse 2024-09-16T08:51:44.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T08:51:44.573 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:44.573 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:44.573 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:44.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T08:51:44.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T08:51:44.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T08:51:44.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:44.574 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:44.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T08:51:44.946 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:45.080 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:45.080 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:45.080 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:51:45.080 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:45.080 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:45.081 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 900 k 2024-09-16T08:51:45.081 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-09-16T08:51:45.081 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T08:51:45.081 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:45.081 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:45.081 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:45.081 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-09-16T08:51:45.081 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:45.082 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 979 k 2024-09-16T08:51:45.082 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.9 M 2024-09-16T08:51:45.082 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:45.312 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 750 kB/s | 80 kB 00:00 2024-09-16T08:51:45.406 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:45.495 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-fuse-19.3.0-4917.g8293d73f.el9.x86_ 3.0 MB/s | 900 kB 00:00 2024-09-16T08:51:45.495 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:45.496 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.3 MB/s | 979 kB 00:00 2024-09-16T08:51:45.507 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:45.529 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:45.529 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:45.539 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:45.540 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:51:45.540 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:45.540 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:45.540 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 154 k 2024-09-16T08:51:45.540 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-09-16T08:51:45.541 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:45.542 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 7.0 M 2024-09-16T08:51:45.542 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 33 M 2024-09-16T08:51:45.542 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:45.610 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:45.610 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:45.804 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:45.948 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T08:51:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 2/2 2024-09-16T08:51:46.047 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T08:51:46.072 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4917.g8293d73f.e 688 kB/s | 154 kB 00:00 2024-09-16T08:51:46.122 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-16T08:51:46.147 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 766 kB/s | 58 kB 00:00 2024-09-16T08:51:46.173 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-16T08:51:46.754 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 2/2 2024-09-16T08:51:46.754 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 1/2 2024-09-16T08:51:47.048 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T08:51:47.049 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:47.049 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:47.049 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T08:51:47.049 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:47.049 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:47.208 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.4 MB/s | 6.0 MB 00:01 2024-09-16T08:51:47.209 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:47.209 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.2 MB/s | 7.0 MB 00:01 2024-09-16T08:51:47.287 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:47.299 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:47.299 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:47.343 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-volume 2024-09-16T08:51:47.400 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:47.701 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:47.876 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T08:51:47.928 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:47.933 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T08:51:48.064 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:48.065 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:48.065 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-09-16T08:51:48.065 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:48.065 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:48.065 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 285 k 2024-09-16T08:51:48.066 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:48.066 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:48.066 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:48.066 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-09-16T08:51:48.066 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:48.066 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 285 k 2024-09-16T08:51:48.066 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.3 M 2024-09-16T08:51:48.066 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:48.308 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-volume-19.3.0-4917.g8293d73f.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-16T08:51:48.308 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:48.309 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-16T08:51:48.309 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:48.317 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:48.317 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:48.356 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:48.356 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:48.465 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T08:51:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T08:51:48.572 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:48.612 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T08:51:48.634 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 6/6 2024-09-16T08:51:48.650 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:48.651 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:48.651 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T08:51:48.651 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:49.416 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 6/6 2024-09-16T08:51:49.416 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 1/6 2024-09-16T08:51:49.416 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T08:51:49.416 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T08:51:49.416 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T08:51:49.417 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T08:51:49.651 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:49.651 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:49.651 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:49.651 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:49.651 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:49.651 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:49.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T08:51:49.695 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:49.695 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:49.695 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:49.695 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T08:51:49.696 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T08:51:49.696 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T08:51:49.696 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T08:51:49.696 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T08:51:49.696 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:49.696 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:49.938 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel 2024-09-16T08:51:49.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-09-16T08:51:50.507 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:50.519 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:50.639 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:50.640 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 900 k 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 979 k 2024-09-16T08:51:50.641 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.9 M 2024-09-16T08:51:50.642 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:50.657 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:50.658 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:50.658 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-09-16T08:51:50.658 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:50.658 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:50.658 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 126 k 2024-09-16T08:51:50.658 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:50.658 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:50.658 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:50.658 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-09-16T08:51:50.659 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:50.659 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 126 k 2024-09-16T08:51:50.659 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 456 k 2024-09-16T08:51:50.659 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:50.817 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 517 kB/s | 80 kB 00:00 2024-09-16T08:51:50.871 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-19.3.0-4917.g8293d73f.el9.x86_64 597 kB/s | 126 kB 00:00 2024-09-16T08:51:50.872 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:50.872 INFO:teuthology.orchestra.run.smithi111.stdout:Total 593 kB/s | 126 kB 00:00 2024-09-16T08:51:50.873 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:50.877 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:50.877 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:50.944 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:50.945 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:50.953 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-fuse-19.3.0-4917.g8293d73f.el9.x86_ 3.1 MB/s | 900 kB 00:00 2024-09-16T08:51:50.954 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:50.954 INFO:teuthology.orchestra.run.smithi114.stdout:Total 3.1 MB/s | 979 kB 00:00 2024-09-16T08:51:50.963 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:50.984 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:50.984 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:51.065 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:51.066 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:51.096 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:51.210 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:51.261 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T08:51:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 2/2 2024-09-16T08:51:51.734 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:51.981 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:51.982 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:51.982 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:51.982 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:52.129 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 2/2 2024-09-16T08:51:52.129 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 1/2 2024-09-16T08:51:52.264 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2 2024-09-16T08:51:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T08:51:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:52.395 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T08:51:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:52.396 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:52.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-volume 2024-09-16T08:51:52.846 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:52.940 INFO:teuthology.orchestra.run.smithi111.stdout:Package libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:52.984 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:52.985 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-09-16T08:51:52.985 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:53.062 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs-devel 2024-09-16T08:51:53.226 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:53.364 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:53.365 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:53.365 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-16T08:51:53.365 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:53.365 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:53.365 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.3.0-4917.g8293d73f.el9 ceph-noarch 285 k 2024-09-16T08:51:53.365 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:53.365 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:53.365 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:53.365 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T08:51:53.366 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:53.366 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 285 k 2024-09-16T08:51:53.366 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 1.3 M 2024-09-16T08:51:53.366 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:53.596 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-volume-19.3.0-4917.g8293d73f.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-16T08:51:53.597 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:53.597 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T08:51:53.597 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:53.606 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:53.606 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:53.633 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:53.639 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:53.639 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:53.766 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:53.767 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:53.767 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-09-16T08:51:53.767 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:53.767 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:51:53.767 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 32 k 2024-09-16T08:51:53.767 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:53.767 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:51:53.767 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:51:53.768 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-09-16T08:51:53.768 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:53.768 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 32 k 2024-09-16T08:51:53.768 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 142 k 2024-09-16T08:51:53.768 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:51:53.855 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:53.924 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs-devel-19.3.0-4917.g8293d73f.el9.x86_6 204 kB/s | 32 kB 00:00 2024-09-16T08:51:53.924 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:53.924 INFO:teuthology.orchestra.run.smithi111.stdout:Total 202 kB/s | 32 kB 00:00 2024-09-16T08:51:53.925 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:51:53.928 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:51:53.928 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:51:53.931 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:53.948 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:53.949 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:51:53.949 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T08:51:53.949 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:53.949 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:51:53.949 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:51:54.035 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:51:54.154 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:54.542 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:54.775 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:54.775 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:54.775 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:51:54.775 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:54.775 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:51:54.775 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:54.963 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2 2024-09-16T08:51:54.969 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.3.0-4917.g8293d73f.el9.noarch 1/1 2024-09-16T08:51:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:54.970 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.3.0-4917.g8293d73f.el9.noarch 2024-09-16T08:51:54.970 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:54.970 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:55.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-09-16T08:51:55.524 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:55.614 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:55.656 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:55.662 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-09-16T08:51:55.662 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:55.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1 2024-09-16T08:51:55.836 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:55.973 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:55.973 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:55.973 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-16T08:51:55.973 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:55.973 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 126 k 2024-09-16T08:51:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:55.974 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:55.974 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:55.974 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T08:51:55.974 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:55.974 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 126 k 2024-09-16T08:51:55.974 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 456 k 2024-09-16T08:51:55.974 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:56.176 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-19.3.0-4917.g8293d73f.el9.x86_64 628 kB/s | 126 kB 00:00 2024-09-16T08:51:56.176 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:56.177 INFO:teuthology.orchestra.run.smithi114.stdout:Total 623 kB/s | 126 kB 00:00 2024-09-16T08:51:56.177 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:56.182 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:56.182 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:56.245 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:56.246 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:56.295 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:56.387 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:56.403 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:56.431 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:56.432 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-09-16T08:51:56.432 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:56.505 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados 2024-09-16T08:51:56.506 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:56.966 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:57.070 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:57.159 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rados-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:57.202 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:57.204 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-09-16T08:51:57.204 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:57.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:57.237 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:57.237 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:51:57.237 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:51:57.238 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:57.238 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:57.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw 2024-09-16T08:51:57.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-09-16T08:51:57.835 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:57.927 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rgw-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:57.969 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:57.970 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-09-16T08:51:57.970 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:58.038 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs 2024-09-16T08:51:58.077 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:58.169 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:58.213 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:58.214 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T08:51:58.214 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:51:58.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-09-16T08:51:58.597 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:58.687 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-cephfs-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:58.730 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:58.732 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-09-16T08:51:58.732 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:58.805 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd 2024-09-16T08:51:58.867 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:59.000 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:51:59.001 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-16T08:51:59.001 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:59.001 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:51:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 32 k 2024-09-16T08:51:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:59.002 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:51:59.002 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:51:59.002 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T08:51:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:51:59.002 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 32 k 2024-09-16T08:51:59.002 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 142 k 2024-09-16T08:51:59.002 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:51:59.154 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-19.3.0-4917.g8293d73f.el9.x86_6 209 kB/s | 32 kB 00:00 2024-09-16T08:51:59.154 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:51:59.155 INFO:teuthology.orchestra.run.smithi114.stdout:Total 207 kB/s | 32 kB 00:00 2024-09-16T08:51:59.155 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:51:59.158 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:51:59.159 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:51:59.184 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:51:59.185 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:51:59.275 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:51:59.370 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:51:59.393 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:51:59.462 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rbd-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:51:59.506 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:51:59.506 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-09-16T08:51:59.507 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:51:59.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse 2024-09-16T08:51:59.758 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:00.001 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:00.002 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:00.002 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:52:00.002 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:52:00.002 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:00.002 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:00.150 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:00.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-09-16T08:52:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:52:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:52:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-09-16T08:52:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:52:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:52:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 85 k 2024-09-16T08:52:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:00.284 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:52:00.284 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:52:00.284 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-09-16T08:52:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:00.284 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 85 k 2024-09-16T08:52:00.284 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 231 k 2024-09-16T08:52:00.285 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:52:00.477 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-19.3.0-4917.g8293d73f.el9.x86_64.rpm 443 kB/s | 85 kB 00:00 2024-09-16T08:52:00.477 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:52:00.478 INFO:teuthology.orchestra.run.smithi111.stdout:Total 440 kB/s | 85 kB 00:00 2024-09-16T08:52:00.478 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:52:00.485 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:52:00.485 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:52:00.549 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:52:00.549 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:52:00.736 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:00.749 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:52:00.826 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:52:00.858 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:00.870 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:52:00.871 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T08:52:00.871 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:00.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-09-16T08:52:01.230 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:01.498 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:52:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:52:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:01.498 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:52:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:01.602 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:52:01.645 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:52:01.646 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T08:52:01.646 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:01.720 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror 2024-09-16T08:52:01.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-09-16T08:52:02.292 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:02.303 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:02.381 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:52:02.425 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:52:02.426 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T08:52:02.426 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:02.444 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:52:02.444 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:52:02.445 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-09-16T08:52:02.445 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:52:02.445 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:52:02.445 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 3.2 M 2024-09-16T08:52:02.445 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:02.445 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:52:02.445 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:52:02.445 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-09-16T08:52:02.445 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:02.446 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 3.2 M 2024-09-16T08:52:02.446 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 14 M 2024-09-16T08:52:02.446 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:52:02.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-09-16T08:52:02.772 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-19.3.0-4917.g8293d73f.el9.x86_64.rpm 9.8 MB/s | 3.2 MB 00:00 2024-09-16T08:52:02.772 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:52:02.773 INFO:teuthology.orchestra.run.smithi111.stdout:Total 9.7 MB/s | 3.2 MB 00:00 2024-09-16T08:52:02.773 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:52:02.782 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:52:02.782 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:52:02.864 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:52:02.864 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:52:03.072 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:03.162 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:52:03.205 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:52:03.206 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T08:52:03.207 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:03.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-09-16T08:52:03.306 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:52:03.372 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:03.401 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:03.401 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:52:03.401 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T08:52:03.401 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T08:52:03.401 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T08:52:03.401 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:03.841 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:03.929 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:52:03.972 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:52:03.973 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T08:52:03.973 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:04.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-09-16T08:52:04.576 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:04.577 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:04.577 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:52:04.577 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:52:04.577 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:04.577 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:52:04.608 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:04.696 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:19.3.0-4917.g8293d73f.el9.x86_64 is already installed. 2024-09-16T08:52:04.739 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:52:04.740 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-16T08:52:04.740 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:04.812 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd 2024-09-16T08:52:04.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-09-16T08:52:05.377 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:05.399 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:05.510 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:52:05.511 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:52:05.511 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 85 k 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 85 k 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 231 k 2024-09-16T08:52:05.512 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:52:05.532 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-09-16T08:52:05.533 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:52:05.533 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-09-16T08:52:05.533 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:52:05.533 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-09-16T08:52:05.533 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 171 k 2024-09-16T08:52:05.533 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:05.533 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-09-16T08:52:05.533 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-09-16T08:52:05.534 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-09-16T08:52:05.534 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:05.534 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 171 k 2024-09-16T08:52:05.534 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 490 k 2024-09-16T08:52:05.534 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-09-16T08:52:05.715 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-19.3.0-4917.g8293d73f.el9.x86_64.rpm 419 kB/s | 85 kB 00:00 2024-09-16T08:52:05.716 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:52:05.716 INFO:teuthology.orchestra.run.smithi114.stdout:Total 417 kB/s | 85 kB 00:00 2024-09-16T08:52:05.716 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:52:05.723 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:52:05.723 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:52:05.741 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-19.3.0-4917.g8293d73f.el9.x86_64.rpm 824 kB/s | 171 kB 00:00 2024-09-16T08:52:05.742 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:52:05.743 INFO:teuthology.orchestra.run.smithi111.stdout:Total 818 kB/s | 171 kB 00:00 2024-09-16T08:52:05.743 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-09-16T08:52:05.750 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-09-16T08:52:05.750 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-09-16T08:52:05.787 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:52:05.787 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:52:05.814 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-09-16T08:52:05.814 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-09-16T08:52:05.967 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:52:05.968 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-09-16T08:52:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:06.073 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:06.456 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:06.545 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:06.736 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:06.736 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:06.736 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:52:06.736 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:52:06.736 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:06.736 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:06.840 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:06.840 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:06.840 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-09-16T08:52:06.840 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:52:06.840 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:52:06.840 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-09-16T08:52:06.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-09-16T08:52:07.081 DEBUG:teuthology.parallel:result is None 2024-09-16T08:52:07.518 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:07.657 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:52:07.659 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:52:07.659 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-16T08:52:07.659 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:52:07.659 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:52:07.659 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 3.2 M 2024-09-16T08:52:07.659 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:07.659 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:52:07.660 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:52:07.660 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T08:52:07.660 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:07.660 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.2 M 2024-09-16T08:52:07.660 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 14 M 2024-09-16T08:52:07.660 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:52:07.969 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-19.3.0-4917.g8293d73f.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-16T08:52:07.969 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:52:07.970 INFO:teuthology.orchestra.run.smithi114.stdout:Total 10 MB/s | 3.2 MB 00:00 2024-09-16T08:52:07.970 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:52:07.979 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:52:07.979 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:52:08.063 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:52:08.063 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:52:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:52:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:08.596 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T08:52:08.596 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T08:52:08.596 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T08:52:08.596 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T08:52:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:09.667 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:52:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:52:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:09.668 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:09.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-09-16T08:52:10.454 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:54 ago on Mon 16 Sep 2024 08:50:16 AM UTC. 2024-09-16T08:52:10.587 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-16T08:52:10.588 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:52:10.588 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-16T08:52:10.588 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:52:10.588 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-16T08:52:10.588 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.3.0-4917.g8293d73f.el9 ceph 171 k 2024-09-16T08:52:10.588 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:10.588 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-16T08:52:10.589 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-16T08:52:10.589 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-16T08:52:10.589 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:10.589 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 171 k 2024-09-16T08:52:10.589 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 490 k 2024-09-16T08:52:10.589 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-16T08:52:10.797 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-19.3.0-4917.g8293d73f.el9.x86_64.rpm 821 kB/s | 171 kB 00:00 2024-09-16T08:52:10.798 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:52:10.798 INFO:teuthology.orchestra.run.smithi114.stdout:Total 815 kB/s | 171 kB 00:00 2024-09-16T08:52:10.798 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-16T08:52:10.806 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-16T08:52:10.806 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-16T08:52:10.870 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-16T08:52:10.870 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-16T08:52:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-16T08:52:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:11.606 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:11.890 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.3.0-4917.g8293d73f.el9.x86_64 1/1 2024-09-16T08:52:11.890 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:11.890 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-16T08:52:11.890 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.3.0-4917.g8293d73f.el9.x86_64 2024-09-16T08:52:11.890 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:52:11.890 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-16T08:52:12.109 DEBUG:teuthology.parallel:result is None 2024-09-16T08:52:12.110 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:52:12.263 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T08:52:12.289 INFO:teuthology.orchestra.run.smithi111.stdout:19.3.0-4917.g8293d73f.el9 2024-09-16T08:52:12.290 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4917.g8293d73f.el9 2024-09-16T08:52:12.290 INFO:teuthology.task.install:The correct ceph version 19.3.0-4917.g8293d73f is installed. 2024-09-16T08:52:12.292 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:52:12.433 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T08:52:12.456 INFO:teuthology.orchestra.run.smithi114.stdout:19.3.0-4917.g8293d73f.el9 2024-09-16T08:52:12.456 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4917.g8293d73f.el9 2024-09-16T08:52:12.456 INFO:teuthology.task.install:The correct ceph version 19.3.0-4917.g8293d73f is installed. 2024-09-16T08:52:12.458 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T08:52:12.458 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:52:12.458 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T08:52:12.489 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:52:12.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T08:52:12.522 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T08:52:12.523 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:52:12.523 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T08:52:12.558 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T08:52:12.626 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:52:12.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T08:52:12.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T08:52:12.718 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T08:52:12.718 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:52:12.719 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T08:52:12.746 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T08:52:12.813 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:52:12.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T08:52:12.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T08:52:12.903 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T08:52:12.903 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:52:12.903 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T08:52:12.929 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T08:52:12.995 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:52:12.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T08:52:13.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T08:52:13.084 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-16T08:52:13.195 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', 'CEPHADM_STRAY_DAEMON'], 'log-only-match': ['CEPHADM_'], 'sha1': '8293d73f8690540e843a81caec373f9cc29cf705', 'cephadm_mode': 'root'} 2024-09-16T08:52:13.195 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:52:13.196 INFO:tasks.cephadm:Cluster fsid is f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:52:13.196 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-16T08:52:13.196 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.111', 'mon.c': '[v2:172.21.15.111:3301,v1:172.21.15.111:6790]', 'mon.b': '172.21.15.114'} 2024-09-16T08:52:13.196 INFO:tasks.cephadm:First mon is mon.a on smithi111 2024-09-16T08:52:13.196 INFO:tasks.cephadm:First mgr is y 2024-09-16T08:52:13.196 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-16T08:52:13.197 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-09-16T08:52:13.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-09-16T08:52:13.248 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-16T08:52:13.248 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:52:13.396 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8293d73f8690540e843a81caec373f9cc29cf705', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83046/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4917-g8293d73f', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4917.g8293d73f'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-13 15:55:14.480698', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-main-2024-09-13-1248', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-16T08:52:13.539 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-guits-main-2024-09-13-1248, sha1 8293d73f8690540e843a81caec373f9cc29cf705 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:52:13.541 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/x86_64/flavors/default/cephadm 2024-09-16T08:52:13.541 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/x86_64/flavors/default/cephadm 2024-09-16T08:52:13.541 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T08:52:13.906 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r--. 1 ubuntu ubuntu 804182 Sep 16 08:52 /home/ubuntu/cephtest/cephadm 2024-09-16T08:52:13.907 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-13-1248/8293d73f8690540e843a81caec373f9cc29cf705/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T08:52:14.254 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 804182 Sep 16 08:52 /home/ubuntu/cephtest/cephadm 2024-09-16T08:52:14.254 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T08:52:14.272 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T08:52:14.301 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 on all hosts... 2024-09-16T08:52:14.301 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 pull 2024-09-16T08:52:14.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 pull 2024-09-16T08:52:15.922 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705... 2024-09-16T08:52:15.949 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705... 2024-09-16T08:52:50.772 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-16T08:52:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.3.0-4917-g8293d73f (8293d73f8690540e843a81caec373f9cc29cf705) squid (dev)", 2024-09-16T08:52:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c", 2024-09-16T08:52:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-09-16T08:52:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce" 2024-09-16T08:52:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-16T08:52:50.773 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-16T08:52:59.100 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-09-16T08:52:59.100 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 19.3.0-4917-g8293d73f (8293d73f8690540e843a81caec373f9cc29cf705) squid (dev)", 2024-09-16T08:52:59.101 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c", 2024-09-16T08:52:59.101 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2024-09-16T08:52:59.101 INFO:teuthology.orchestra.run.smithi111.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce" 2024-09-16T08:52:59.101 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-09-16T08:52:59.101 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-09-16T08:52:59.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-09-16T08:52:59.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-09-16T08:52:59.206 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-09-16T08:52:59.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-09-16T08:52:59.276 INFO:tasks.cephadm:Writing seed config... 2024-09-16T08:52:59.277 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-16T08:52:59.277 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-16T08:52:59.277 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-16T08:52:59.277 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-16T08:52:59.277 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-16T08:52:59.278 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-16T08:52:59.278 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-16T08:52:59.278 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-16T08:52:59.278 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-16T08:52:59.278 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-16T08:52:59.278 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-16T08:52:59.278 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-09-16T08:52:59.278 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-09-16T08:52:59.278 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-16T08:52:59.279 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:52:59.279 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-16T08:52:59.295 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f774c998-7408-11ef-bceb-c7b262605968 mon election default strategy = 3 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-09-16T08:52:59.296 DEBUG:teuthology.orchestra.run.smithi111:mon.a> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.a.service 2024-09-16T08:52:59.338 DEBUG:teuthology.orchestra.run.smithi111:mgr.y> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.y.service 2024-09-16T08:52:59.381 INFO:tasks.cephadm:Bootstrapping... 2024-09-16T08:52:59.382 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 -v bootstrap --fsid f774c998-7408-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.111 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-16T08:52:59.585 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-09-16T08:52:59.585 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705', '-v', 'bootstrap', '--fsid', 'f774c998-7408-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.111', '--skip-admin-label'] 2024-09-16T08:52:59.585 INFO:teuthology.orchestra.run.smithi111.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-16T08:52:59.585 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying podman|docker is present... 2024-09-16T08:52:59.609 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 5.2.2 2024-09-16T08:52:59.609 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying lvm2 is present... 2024-09-16T08:52:59.609 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying time synchronization is in place... 2024-09-16T08:52:59.617 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T08:52:59.617 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T08:52:59.624 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T08:52:59.624 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-09-16T08:52:59.630 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout enabled 2024-09-16T08:52:59.637 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout active 2024-09-16T08:52:59.637 INFO:teuthology.orchestra.run.smithi111.stdout:Unit chronyd.service is enabled and running 2024-09-16T08:52:59.637 INFO:teuthology.orchestra.run.smithi111.stdout:Repeating the final host check... 2024-09-16T08:52:59.659 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 5.2.2 2024-09-16T08:52:59.659 INFO:teuthology.orchestra.run.smithi111.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-16T08:52:59.659 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl is present 2024-09-16T08:52:59.659 INFO:teuthology.orchestra.run.smithi111.stdout:lvcreate is present 2024-09-16T08:52:59.665 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T08:52:59.665 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T08:52:59.671 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T08:52:59.671 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-09-16T08:52:59.679 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout enabled 2024-09-16T08:52:59.685 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout active 2024-09-16T08:52:59.686 INFO:teuthology.orchestra.run.smithi111.stdout:Unit chronyd.service is enabled and running 2024-09-16T08:52:59.686 INFO:teuthology.orchestra.run.smithi111.stdout:Host looks OK 2024-09-16T08:52:59.686 INFO:teuthology.orchestra.run.smithi111.stdout:Cluster fsid: f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:52:59.686 INFO:teuthology.orchestra.run.smithi111.stdout:Acquiring lock 139733718162064 on /run/cephadm/f774c998-7408-11ef-bceb-c7b262605968.lock 2024-09-16T08:52:59.686 INFO:teuthology.orchestra.run.smithi111.stdout:Lock 139733718162064 acquired on /run/cephadm/f774c998-7408-11ef-bceb-c7b262605968.lock 2024-09-16T08:52:59.686 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying IP 172.21.15.111 port 3300 ... 2024-09-16T08:52:59.687 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying IP 172.21.15.111 port 6789 ... 2024-09-16T08:52:59.687 INFO:teuthology.orchestra.run.smithi111.stdout:Base mon IP(s) is [172.21.15.111:3300, 172.21.15.111:6789], mon addrv is [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2024-09-16T08:52:59.689 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.111 metric 100 2024-09-16T08:52:59.689 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.111 metric 100 2024-09-16T08:52:59.691 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-16T08:52:59.691 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-16T08:52:59.691 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-16T08:52:59.694 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-16T08:52:59.694 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-16T08:52:59.694 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T08:52:59.694 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-16T08:52:59.694 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link noprefixroute 2024-09-16T08:52:59.694 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T08:52:59.694 INFO:teuthology.orchestra.run.smithi111.stdout:Mon IP `172.21.15.111` is in CIDR network `172.21.0.0/20` 2024-09-16T08:52:59.694 INFO:teuthology.orchestra.run.smithi111.stdout:Mon IP `172.21.15.111` is in CIDR network `172.21.0.0/20` 2024-09-16T08:52:59.695 INFO:teuthology.orchestra.run.smithi111.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-16T08:52:59.695 INFO:teuthology.orchestra.run.smithi111.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-16T08:52:59.695 INFO:teuthology.orchestra.run.smithi111.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705... 2024-09-16T08:53:00.162 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c 2024-09-16T08:53:00.162 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705... 2024-09-16T08:53:00.162 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Getting image source signatures 2024-09-16T08:53:00.162 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Copying blob sha256:7b1291ae133cf24c065f94db08f06624ab089b121831fa7f7f15d46063d39776 2024-09-16T08:53:00.162 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-16T08:53:00.163 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Copying config sha256:425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c 2024-09-16T08:53:00.163 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-16T08:53:01.108 INFO:teuthology.orchestra.run.smithi111.stdout:ceph: stdout ceph version 19.3.0-4917-g8293d73f (8293d73f8690540e843a81caec373f9cc29cf705) squid (dev) 2024-09-16T08:53:01.108 INFO:teuthology.orchestra.run.smithi111.stdout:Ceph version: ceph version 19.3.0-4917-g8293d73f (8293d73f8690540e843a81caec373f9cc29cf705) squid (dev) 2024-09-16T08:53:01.109 INFO:teuthology.orchestra.run.smithi111.stdout:Extracting ceph user uid/gid from container image... 2024-09-16T08:53:02.089 INFO:teuthology.orchestra.run.smithi111.stdout:stat: stdout 167 167 2024-09-16T08:53:02.089 INFO:teuthology.orchestra.run.smithi111.stdout:Creating initial keys... 2024-09-16T08:53:03.069 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph-authtool: stdout AQDu8edmoUUyGhAAFWN+71E0xhPGPvB4Wbw8ug== 2024-09-16T08:53:04.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph-authtool: stdout AQDv8edmZKqfGBAA8oxl87ixQgulX/UiKtKw9g== 2024-09-16T08:53:05.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph-authtool: stdout AQDw8edm3Y5zFxAAbIuXOzOt71JX/MMARi3U0Q== 2024-09-16T08:53:05.090 INFO:teuthology.orchestra.run.smithi111.stdout:Creating initial monmap... 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool for a [v2:172.21.15.111:3300,v1:172.21.15.111:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout:setting min_mon_release = quincy 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: set fsid to f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:53:05.997 INFO:teuthology.orchestra.run.smithi111.stdout:Creating mon... 2024-09-16T08:53:06.935 INFO:teuthology.orchestra.run.smithi111.stdout:create mon.a on 2024-09-16T08:53:07.153 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-16T08:53:07.300 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-16T08:53:07.475 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f774c998-7408-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-f774c998-7408-11ef-bceb-c7b262605968.target. 2024-09-16T08:53:07.475 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-f774c998-7408-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-f774c998-7408-11ef-bceb-c7b262605968.target. 2024-09-16T08:53:07.689 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.a 2024-09-16T08:53:07.689 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-16T08:53:07.853 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f774c998-7408-11ef-bceb-c7b262605968.target.wants/ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-f774c998-7408-11ef-bceb-c7b262605968@.service. 2024-09-16T08:53:08.380 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:08 smithi111 ceph-mon[29329]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T08:53:08.380 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:08 smithi111 bash[29290]: cc9b5daf119cfb8fbbfa564e221ea9ba2c82016dccd061d8bb6046fecc432ba9 2024-09-16T08:53:08.393 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T08:53:08.393 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2024-09-16T08:53:08.400 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T08:53:08.400 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-09-16T08:53:08.400 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2024-09-16T08:53:08.400 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T08:53:08.400 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mon to start... 2024-09-16T08:53:08.401 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mon... 2024-09-16T08:53:08.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:08 smithi111 systemd[1]: Started Ceph mon.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:53:09.419 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout cluster: 2024-09-16T08:53:09.419 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout id: f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:09.419 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-16T08:53:09.419 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:09.419 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout services: 2024-09-16T08:53:09.419 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.57043s) 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout data: 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout pgs: 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.811+0000 7f64d5e22640 1 Processor -- start 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.812+0000 7f64d5e22640 1 -- start start 2024-09-16T08:53:09.420 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.812+0000 7f64d5e22640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d0106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.812+0000 7f64d5e22640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f64d0107000 con 0x7f64d0106610 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.812+0000 7f64cf7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d0106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.812+0000 7f64cf7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d0106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35500/0 (socket says 172.21.15.111:35500) 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.812+0000 7f64cf7fe640 1 -- 172.21.15.111:0/1855614243 learned_addr learned my addr 172.21.15.111:0/1855614243 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.812+0000 7f64cf7fe640 1 -- 172.21.15.111:0/1855614243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d0107180 con 0x7f64d0106610 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.814+0000 7f64cf7fe640 1 --2- 172.21.15.111:0/1855614243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d0106a10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009820 tx=0x7f64c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2447c5ea37069c27 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64ceffd640 1 -- 172.21.15.111:0/1855614243 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64c0037070 con 0x7f64d0106610 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64ceffd640 1 -- 172.21.15.111:0/1855614243 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c0032440 con 0x7f64d0106610 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64ceffd640 1 -- 172.21.15.111:0/1855614243 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64c0032740 con 0x7f64d0106610 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64d5e22640 1 -- 172.21.15.111:0/1855614243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 msgr2=0x7f64d0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64d5e22640 1 --2- 172.21.15.111:0/1855614243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d0106a10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009820 tx=0x7f64c002a270 comp rx=0 tx=0).stop 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64d5e22640 1 -- 172.21.15.111:0/1855614243 shutdown_connections 2024-09-16T08:53:09.421 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64d5e22640 1 --2- 172.21.15.111:0/1855614243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d0106a10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64d5e22640 1 -- 172.21.15.111:0/1855614243 >> 172.21.15.111:0/1855614243 conn(0x7f64d0101f00 msgr2=0x7f64d0104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64d5e22640 1 -- 172.21.15.111:0/1855614243 shutdown_connections 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.815+0000 7f64d5e22640 1 -- 172.21.15.111:0/1855614243 wait complete. 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.816+0000 7f64d5e22640 1 Processor -- start 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.816+0000 7f64d5e22640 1 -- start start 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.816+0000 7f64d5e22640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d007c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.816+0000 7f64d5e22640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f64d0107b90 con 0x7f64d0106610 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.816+0000 7f64cf7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d007c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.816+0000 7f64cf7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d007c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35502/0 (socket says 172.21.15.111:35502) 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.816+0000 7f64cf7fe640 1 -- 172.21.15.111:0/801512130 learned_addr learned my addr 172.21.15.111:0/801512130 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:09.422 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.817+0000 7f64cf7fe640 1 -- 172.21.15.111:0/801512130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d007c6e0 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.817+0000 7f64cf7fe640 1 --2- 172.21.15.111:0/801512130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d007c180 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f64c00090b0 tx=0x7f64c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.817+0000 7f64cd7fa640 1 -- 172.21.15.111:0/801512130 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64c0041070 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.817+0000 7f64cd7fa640 1 -- 172.21.15.111:0/801512130 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c003c7a0 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.817+0000 7f64d5e22640 1 -- 172.21.15.111:0/801512130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d007aad0 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.817+0000 7f64cd7fa640 1 -- 172.21.15.111:0/801512130 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64c0037040 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.817+0000 7f64d5e22640 1 -- 172.21.15.111:0/801512130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d007b050 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.817+0000 7f64cd7fa640 1 -- 172.21.15.111:0/801512130 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f64c0044070 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.817+0000 7f64cd7fa640 1 -- 172.21.15.111:0/801512130 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f64c003d3f0 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.818+0000 7f64d5e22640 1 -- 172.21.15.111:0/801512130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d0108340 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.820+0000 7f64cd7fa640 1 -- 172.21.15.111:0/801512130 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f64c0046450 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.881+0000 7f64d5e22640 1 -- 172.21.15.111:0/801512130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f64d007b500 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.882+0000 7f64cd7fa640 1 -- 172.21.15.111:0/801512130 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f64c0046630 con 0x7f64d0106610 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.882+0000 7f64d5e22640 1 -- 172.21.15.111:0/801512130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 msgr2=0x7f64d007c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:09.423 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.882+0000 7f64d5e22640 1 --2- 172.21.15.111:0/801512130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d007c180 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f64c00090b0 tx=0x7f64c00046c0 comp rx=0 tx=0).stop 2024-09-16T08:53:09.424 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.883+0000 7f64d5e22640 1 -- 172.21.15.111:0/801512130 shutdown_connections 2024-09-16T08:53:09.424 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.883+0000 7f64d5e22640 1 --2- 172.21.15.111:0/801512130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64d0106610 0x7f64d007c180 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:09.424 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.883+0000 7f64d5e22640 1 -- 172.21.15.111:0/801512130 >> 172.21.15.111:0/801512130 conn(0x7f64d0101f00 msgr2=0x7f64d018e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:09.424 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.883+0000 7f64d5e22640 1 -- 172.21.15.111:0/801512130 shutdown_connections 2024-09-16T08:53:09.424 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:08.883+0000 7f64d5e22640 1 -- 172.21.15.111:0/801512130 wait complete. 2024-09-16T08:53:09.424 INFO:teuthology.orchestra.run.smithi111.stdout:mon is available 2024-09-16T08:53:09.424 INFO:teuthology.orchestra.run.smithi111.stdout:Assimilating anything we can from ceph.conf... 2024-09-16T08:53:09.545 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T08:53:09.545 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: monmap epoch 1 2024-09-16T08:53:09.545 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:09.545 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: last_changed 2024-09-16T08:53:05.472892+0000 2024-09-16T08:53:09.545 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: created 2024-09-16T08:53:05.472892+0000 2024-09-16T08:53:09.546 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: min_mon_release 19 (squid) 2024-09-16T08:53:09.546 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: election_strategy: 1 2024-09-16T08:53:09.546 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: 0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2024-09-16T08:53:09.546 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: fsmap 2024-09-16T08:53:09.546 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: osdmap e1: 0 total, 0 up, 0 in 2024-09-16T08:53:09.546 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: mgrmap e1: no daemons active 2024-09-16T08:53:09.546 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:09 smithi111 ceph-mon[29329]: from='client.? 172.21.15.111:0/801512130' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-16T08:53:10.539 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:10.539 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T08:53:10.539 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout fsid = f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:10.539 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-16T08:53:10.540 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2024-09-16T08:53:10.540 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T08:53:10.540 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T08:53:10.540 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T08:53:10.541 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T08:53:10.541 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:10.541 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T08:53:10.541 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T08:53:10.541 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:10.542 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T08:53:10.542 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T08:53:10.542 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T08:53:10.542 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.874+0000 7f526b3a3640 1 Processor -- start 2024-09-16T08:53:10.542 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.874+0000 7f526b3a3640 1 -- start start 2024-09-16T08:53:10.542 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.875+0000 7f526b3a3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f5264106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:10.543 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.875+0000 7f526b3a3640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5264107000 con 0x7f5264106610 2024-09-16T08:53:10.543 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.875+0000 7f526913f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f5264106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:10.543 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.875+0000 7f526913f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f5264106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44972/0 (socket says 172.21.15.111:44972) 2024-09-16T08:53:10.543 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.875+0000 7f526913f640 1 -- 172.21.15.111:0/3134935787 learned_addr learned my addr 172.21.15.111:0/3134935787 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:10.543 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.875+0000 7f526913f640 1 -- 172.21.15.111:0/3134935787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5264107180 con 0x7f5264106610 2024-09-16T08:53:10.543 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.876+0000 7f526913f640 1 --2- 172.21.15.111:0/3134935787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f5264106a10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5260009820 tx=0x7f526002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43649658f4044c74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:10.544 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.876+0000 7f525bfff640 1 -- 172.21.15.111:0/3134935787 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5260037070 con 0x7f5264106610 2024-09-16T08:53:10.544 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.876+0000 7f525bfff640 1 -- 172.21.15.111:0/3134935787 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5260032440 con 0x7f5264106610 2024-09-16T08:53:10.544 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.876+0000 7f525bfff640 1 -- 172.21.15.111:0/3134935787 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5260032740 con 0x7f5264106610 2024-09-16T08:53:10.544 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 -- 172.21.15.111:0/3134935787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 msgr2=0x7f5264106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:10.544 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 --2- 172.21.15.111:0/3134935787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f5264106a10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5260009820 tx=0x7f526002a270 comp rx=0 tx=0).stop 2024-09-16T08:53:10.545 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 -- 172.21.15.111:0/3134935787 shutdown_connections 2024-09-16T08:53:10.545 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 --2- 172.21.15.111:0/3134935787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f5264106a10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:10.545 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 -- 172.21.15.111:0/3134935787 >> 172.21.15.111:0/3134935787 conn(0x7f5264101f00 msgr2=0x7f5264104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:10.546 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 -- 172.21.15.111:0/3134935787 shutdown_connections 2024-09-16T08:53:10.546 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 -- 172.21.15.111:0/3134935787 wait complete. 2024-09-16T08:53:10.546 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 Processor -- start 2024-09-16T08:53:10.546 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 -- start start 2024-09-16T08:53:10.546 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f526407c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:10.547 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526b3a3640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5264107b90 con 0x7f5264106610 2024-09-16T08:53:10.547 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.877+0000 7f526913f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f526407c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:10.547 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f526913f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f526407c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44974/0 (socket says 172.21.15.111:44974) 2024-09-16T08:53:10.547 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f526913f640 1 -- 172.21.15.111:0/804133355 learned_addr learned my addr 172.21.15.111:0/804133355 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:10.547 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f526913f640 1 -- 172.21.15.111:0/804133355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526407c6e0 con 0x7f5264106610 2024-09-16T08:53:10.547 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f526913f640 1 --2- 172.21.15.111:0/804133355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f526407c180 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f52600090b0 tx=0x7f52600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:10.548 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f525a7fc640 1 -- 172.21.15.111:0/804133355 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5260041070 con 0x7f5264106610 2024-09-16T08:53:10.548 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f525a7fc640 1 -- 172.21.15.111:0/804133355 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526003c7a0 con 0x7f5264106610 2024-09-16T08:53:10.548 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f525a7fc640 1 -- 172.21.15.111:0/804133355 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5260037040 con 0x7f5264106610 2024-09-16T08:53:10.548 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f526b3a3640 1 -- 172.21.15.111:0/804133355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526407aad0 con 0x7f5264106610 2024-09-16T08:53:10.548 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f526b3a3640 1 -- 172.21.15.111:0/804133355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f526407b050 con 0x7f5264106610 2024-09-16T08:53:10.548 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f525a7fc640 1 -- 172.21.15.111:0/804133355 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5260044070 con 0x7f5264106610 2024-09-16T08:53:10.549 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.878+0000 7f525a7fc640 1 -- 172.21.15.111:0/804133355 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f526003d3f0 con 0x7f5264106610 2024-09-16T08:53:10.549 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.879+0000 7f526b3a3640 1 -- 172.21.15.111:0/804133355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5264108340 con 0x7f5264106610 2024-09-16T08:53:10.549 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.881+0000 7f525a7fc640 1 -- 172.21.15.111:0/804133355 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f5260046450 con 0x7f5264106610 2024-09-16T08:53:10.550 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.940+0000 7f526b3a3640 1 -- 172.21.15.111:0/804133355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f526407b500 con 0x7f5264106610 2024-09-16T08:53:10.550 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.942+0000 7f525a7fc640 1 -- 172.21.15.111:0/804133355 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+376 (secure 0 0 0) 0x7f5260046630 con 0x7f5264106610 2024-09-16T08:53:10.550 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.943+0000 7f525a7fc640 1 -- 172.21.15.111:0/804133355 <== mon.0 v2:172.21.15.111:3300/0 8 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f5260056070 con 0x7f5264106610 2024-09-16T08:53:10.550 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.944+0000 7f526b3a3640 1 -- 172.21.15.111:0/804133355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 msgr2=0x7f526407c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:10.550 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.944+0000 7f526b3a3640 1 --2- 172.21.15.111:0/804133355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f526407c180 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f52600090b0 tx=0x7f52600046c0 comp rx=0 tx=0).stop 2024-09-16T08:53:10.551 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.944+0000 7f526b3a3640 1 -- 172.21.15.111:0/804133355 shutdown_connections 2024-09-16T08:53:10.551 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.944+0000 7f526b3a3640 1 --2- 172.21.15.111:0/804133355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5264106610 0x7f526407c180 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:10.551 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.944+0000 7f526b3a3640 1 -- 172.21.15.111:0/804133355 >> 172.21.15.111:0/804133355 conn(0x7f5264101f00 msgr2=0x7f526418e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:10.551 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.944+0000 7f526b3a3640 1 -- 172.21.15.111:0/804133355 shutdown_connections 2024-09-16T08:53:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:09.944+0000 7f526b3a3640 1 -- 172.21.15.111:0/804133355 wait complete. 2024-09-16T08:53:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:Generating new minimal ceph.conf... 2024-09-16T08:53:10.652 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:10 smithi111 ceph-mon[29329]: from='client.? 172.21.15.111:0/804133355' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-16T08:53:10.652 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:10 smithi111 ceph-mon[29329]: from='client.? 172.21.15.111:0/804133355' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-16T08:53:11.566 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:11 smithi111 ceph-mon[29329]: from='client.? 172.21.15.111:0/1919629644' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:53:11.572 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.922+0000 7f43d68a0640 1 Processor -- start 2024-09-16T08:53:11.572 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.922+0000 7f43d68a0640 1 -- start start 2024-09-16T08:53:11.572 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.923+0000 7f43d68a0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d01091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.923+0000 7f43d68a0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f43d0109700 con 0x7f43d0079b80 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.923+0000 7f43cffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d01091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.923+0000 7f43cffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d01091c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44980/0 (socket says 172.21.15.111:44980) 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.923+0000 7f43cffff640 1 -- 172.21.15.111:0/3447901972 learned_addr learned my addr 172.21.15.111:0/3447901972 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.923+0000 7f43cffff640 1 -- 172.21.15.111:0/3447901972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d0109880 con 0x7f43d0079b80 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.924+0000 7f43cffff640 1 --2- 172.21.15.111:0/3447901972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d01091c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009820 tx=0x7f43b802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866a739653ed80c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.924+0000 7f43ceffd640 1 -- 172.21.15.111:0/3447901972 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43b8037070 con 0x7f43d0079b80 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.924+0000 7f43ceffd640 1 -- 172.21.15.111:0/3447901972 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f43b8032870 con 0x7f43d0079b80 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.924+0000 7f43ceffd640 1 -- 172.21.15.111:0/3447901972 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43b80305d0 con 0x7f43d0079b80 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.925+0000 7f43d68a0640 1 -- 172.21.15.111:0/3447901972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 msgr2=0x7f43d01091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.925+0000 7f43d68a0640 1 --2- 172.21.15.111:0/3447901972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d01091c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009820 tx=0x7f43b802a270 comp rx=0 tx=0).stop 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.925+0000 7f43d68a0640 1 -- 172.21.15.111:0/3447901972 shutdown_connections 2024-09-16T08:53:11.573 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.925+0000 7f43d68a0640 1 --2- 172.21.15.111:0/3447901972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d01091c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.925+0000 7f43d68a0640 1 -- 172.21.15.111:0/3447901972 >> 172.21.15.111:0/3447901972 conn(0x7f43d0100df0 msgr2=0x7f43d0103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.925+0000 7f43d68a0640 1 -- 172.21.15.111:0/3447901972 shutdown_connections 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.925+0000 7f43d68a0640 1 -- 172.21.15.111:0/3447901972 wait complete. 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.925+0000 7f43d68a0640 1 Processor -- start 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.925+0000 7f43d68a0640 1 -- start start 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43d68a0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43d68a0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f43d010a8a0 con 0x7f43d0079b80 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d019a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44990/0 (socket says 172.21.15.111:44990) 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cffff640 1 -- 172.21.15.111:0/1919629644 learned_addr learned my addr 172.21.15.111:0/1919629644 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cffff640 1 -- 172.21.15.111:0/1919629644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d019abe0 con 0x7f43d0079b80 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cffff640 1 --2- 172.21.15.111:0/1919629644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d019a680 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f43b80097f0 tx=0x7f43b8030b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cd7fa640 1 -- 172.21.15.111:0/1919629644 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43b803f020 con 0x7f43d0079b80 2024-09-16T08:53:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cd7fa640 1 -- 172.21.15.111:0/1919629644 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f43b8030dc0 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cd7fa640 1 -- 172.21.15.111:0/1919629644 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43b8037040 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43d68a0640 1 -- 172.21.15.111:0/1919629644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d019ae70 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43d68a0640 1 -- 172.21.15.111:0/1919629644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d019b830 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cd7fa640 1 -- 172.21.15.111:0/1919629644 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f43b8042020 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.926+0000 7f43cd7fa640 1 -- 172.21.15.111:0/1919629644 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f43b8044930 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.927+0000 7f43d68a0640 1 -- 172.21.15.111:0/1919629644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d0107230 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.929+0000 7f43cd7fa640 1 -- 172.21.15.111:0/1919629644 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f43b8049020 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.988+0000 7f43d68a0640 1 -- 172.21.15.111:0/1919629644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f43d019bd80 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.988+0000 7f43cd7fa640 1 -- 172.21.15.111:0/1919629644 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f43b8044400 con 0x7f43d0079b80 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.989+0000 7f43d68a0640 1 -- 172.21.15.111:0/1919629644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 msgr2=0x7f43d019a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.989+0000 7f43d68a0640 1 --2- 172.21.15.111:0/1919629644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d019a680 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f43b80097f0 tx=0x7f43b8030b00 comp rx=0 tx=0).stop 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.989+0000 7f43d68a0640 1 -- 172.21.15.111:0/1919629644 shutdown_connections 2024-09-16T08:53:11.575 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.989+0000 7f43d68a0640 1 --2- 172.21.15.111:0/1919629644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d0079b80 0x7f43d019a680 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:11.576 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.989+0000 7f43d68a0640 1 -- 172.21.15.111:0/1919629644 >> 172.21.15.111:0/1919629644 conn(0x7f43d0100df0 msgr2=0x7f43d0101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:11.576 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.989+0000 7f43d68a0640 1 -- 172.21.15.111:0/1919629644 shutdown_connections 2024-09-16T08:53:11.576 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:10.989+0000 7f43d68a0640 1 -- 172.21.15.111:0/1919629644 wait complete. 2024-09-16T08:53:11.576 INFO:teuthology.orchestra.run.smithi111.stdout:Restarting the monitor... 2024-09-16T08:53:11.846 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:11 smithi111 systemd[1]: Stopping Ceph mon.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T08:53:12.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:11 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a[29306]: 2024-09-16T08:53:11.844+0000 7fb900bb1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T08:53:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:11 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a[29306]: 2024-09-16T08:53:11.844+0000 7fb900bb1640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-16T08:53:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:12 smithi111 podman[29650]: 2024-09-16 08:53:12.008473316 +0000 UTC m=+0.275797497 container died cc9b5daf119cfb8fbbfa564e221ea9ba2c82016dccd061d8bb6046fecc432ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T08:53:12.504 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:12 smithi111 podman[29650]: 2024-09-16 08:53:12.309990318 +0000 UTC m=+0.577314496 container cleanup cc9b5daf119cfb8fbbfa564e221ea9ba2c82016dccd061d8bb6046fecc432ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T08:53:12.504 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:12 smithi111 bash[29650]: ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a 2024-09-16T08:53:12.504 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:12 smithi111 podman[29667]: 2024-09-16 08:53:12.502661148 +0000 UTC m=+0.492058420 container remove cc9b5daf119cfb8fbbfa564e221ea9ba2c82016dccd061d8bb6046fecc432ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-16T08:53:12.853 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:12 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-16T08:53:12.853 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:12 smithi111 systemd[1]: Stopped Ceph mon.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:53:12.853 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:12 smithi111 systemd[1]: Starting Ceph mon.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T08:53:13.101 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:12 smithi111 podman[29766]: 2024-09-16 08:53:12.752736611 +0000 UTC m=+0.015225026 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:53:13.102 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:12 smithi111 podman[29766]: 2024-09-16 08:53:12.861739165 +0000 UTC m=+0.124227576 container create 676e47c984e2ea0d1182324f64bd7114a8677a1ebc2178b05fd635f520f22dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-16T08:53:13.102 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 podman[29766]: 2024-09-16 08:53:13.072509394 +0000 UTC m=+0.334997804 container init 676e47c984e2ea0d1182324f64bd7114a8677a1ebc2178b05fd635f520f22dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-16T08:53:13.102 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 podman[29766]: 2024-09-16 08:53:13.077093981 +0000 UTC m=+0.339582392 container start 676e47c984e2ea0d1182324f64bd7114a8677a1ebc2178b05fd635f520f22dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-16T08:53:13.102 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: set uid:gid to 167:167 (ceph:ceph) 2024-09-16T08:53:13.102 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: ceph version 19.3.0-4917-g8293d73f (8293d73f8690540e843a81caec373f9cc29cf705) squid (dev), process ceph-mon, pid 7 2024-09-16T08:53:13.102 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: pidfile_write: ignore empty --pid-file 2024-09-16T08:53:13.102 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: load: jerasure load: lrc 2024-09-16T08:53:13.160 INFO:teuthology.orchestra.run.smithi111.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-16T08:53:13.377 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: RocksDB version: 7.9.2 2024-09-16T08:53:13.377 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Git sha 0 2024-09-16T08:53:13.377 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Compile date 2024-09-13 14:50:41 2024-09-16T08:53:13.377 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: DB SUMMARY 2024-09-16T08:53:13.377 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: DB Session ID: VZLHX7RRL97P5E3G2IV4 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: CURRENT file: CURRENT 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: IDENTITY file: IDENTITY 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 91022 ; 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.error_if_exists: 0 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.create_if_missing: 0 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.paranoid_checks: 1 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.env: 0x55730853b120 2024-09-16T08:53:13.378 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.fs: PosixFileSystem 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.info_log: 0x55730ac05700 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_file_opening_threads: 16 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.statistics: (nil) 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.use_fsync: 0 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_log_file_size: 0 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.keep_log_file_num: 1000 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.recycle_log_file_num: 0 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.allow_fallocate: 1 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.allow_mmap_reads: 0 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.allow_mmap_writes: 0 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.use_direct_reads: 0 2024-09-16T08:53:13.379 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-16T08:53:13.380 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.create_missing_column_families: 0 2024-09-16T08:53:13.380 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.db_log_dir: 2024-09-16T08:53:13.380 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.wal_dir: 2024-09-16T08:53:13.380 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-16T08:53:13.380 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-16T08:53:13.380 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-16T08:53:13.380 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-16T08:53:13.381 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-16T08:53:13.381 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.advise_random_on_open: 1 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.db_write_buffer_size: 0 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.write_buffer_manager: 0x55730ac09b80 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.rate_limiter: (nil) 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.wal_recovery_mode: 2 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.enable_thread_tracking: 0 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.enable_pipelined_write: 0 2024-09-16T08:53:13.382 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.unordered_write: 0 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.row_cache: None 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.wal_filter: None 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.allow_ingest_behind: 0 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.two_write_queues: 0 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.manual_wal_flush: 0 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.wal_compression: 0 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.atomic_flush: 0 2024-09-16T08:53:13.383 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-16T08:53:13.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-16T08:53:13.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-16T08:53:13.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.log_readahead_size: 0 2024-09-16T08:53:13.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-16T08:53:13.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.best_efforts_recovery: 0 2024-09-16T08:53:13.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-16T08:53:13.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-16T08:53:13.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.allow_data_in_errors: 0 2024-09-16T08:53:13.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.db_host_id: __hostname__ 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_background_jobs: 2 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_background_compactions: -1 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_subcompactions: 1 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_total_wal_size: 0 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_open_files: -1 2024-09-16T08:53:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bytes_per_sync: 0 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_readahead_size: 0 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_background_flushes: -1 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Compression algorithms supported: 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: kZSTD supported: 0 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: kXpressCompression supported: 0 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: kBZip2Compression supported: 0 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: kLZ4Compression supported: 1 2024-09-16T08:53:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: kZlibCompression supported: 1 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: kLZ4HCCompression supported: 1 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: kSnappyCompression supported: 1 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.merge_operator: 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_filter: None 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_filter_factory: None 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.sst_partitioner_factory: None 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-16T08:53:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55730ac05320) 2024-09-16T08:53:13.388 INFO:journalctl@ceph.mon.a.smithi111.stdout: cache_index_and_filter_blocks: 1 2024-09-16T08:53:13.388 INFO:journalctl@ceph.mon.a.smithi111.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-16T08:53:13.388 INFO:journalctl@ceph.mon.a.smithi111.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-16T08:53:13.388 INFO:journalctl@ceph.mon.a.smithi111.stdout: pin_top_level_index_and_filter: 1 2024-09-16T08:53:13.388 INFO:journalctl@ceph.mon.a.smithi111.stdout: index_type: 0 2024-09-16T08:53:13.388 INFO:journalctl@ceph.mon.a.smithi111.stdout: data_block_index_type: 0 2024-09-16T08:53:13.388 INFO:journalctl@ceph.mon.a.smithi111.stdout: index_shortening: 1 2024-09-16T08:53:13.388 INFO:journalctl@ceph.mon.a.smithi111.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-16T08:53:13.388 INFO:journalctl@ceph.mon.a.smithi111.stdout: checksum: 4 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: no_block_cache: 0 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_cache: 0x55730abf9610 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_cache_name: BinnedLRUCache 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_cache_options: 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: capacity : 536870912 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: num_shard_bits : 4 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: strict_capacity_limit : 0 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: high_pri_pool_ratio: 0.000 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_cache_compressed: (nil) 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: persistent_cache: (nil) 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_size: 4096 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_size_deviation: 10 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_restart_interval: 16 2024-09-16T08:53:13.389 INFO:journalctl@ceph.mon.a.smithi111.stdout: index_block_restart_interval: 1 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: metadata_block_size: 4096 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: partition_filters: 0 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: use_delta_encoding: 1 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: filter_policy: bloomfilter 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: whole_key_filtering: 1 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: verify_compression: 0 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: read_amp_bytes_per_bit: 0 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: format_version: 5 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: enable_index_compression: 1 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_align: 0 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: max_auto_readahead_size: 262144 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: prepopulate_block_cache: 0 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: initial_auto_readahead_size: 8192 2024-09-16T08:53:13.390 INFO:journalctl@ceph.mon.a.smithi111.stdout: num_file_reads_for_auto_readahead: 2 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.write_buffer_size: 33554432 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_write_buffer_number: 2 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression: NoCompression 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression: Disabled 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.prefix_extractor: nullptr 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.num_levels: 7 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-16T08:53:13.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression_opts.level: 32767 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression_opts.strategy: 0 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-16T08:53:13.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression_opts.enabled: false 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.target_file_size_base: 67108864 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-16T08:53:13.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.arena_block_size: 1048576 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.disable_auto_compactions: 0 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-16T08:53:13.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.inplace_update_support: 0 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-16T08:53:13.395 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.bloom_locality: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.max_successive_merges: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.paranoid_file_checks: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.force_consistency_checks: 1 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.report_bg_io_stats: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.ttl: 2592000 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.enable_blob_files: false 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.min_blob_size: 0 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.blob_file_size: 268435456 2024-09-16T08:53:13.396 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.blob_file_starting_level: 0 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e2cb7dc7-eca3-4260-b4f6-b7f56eb2a9a3 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726476793103449, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726476793104286, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 87487, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 269, "table_properties": {"data_size": 85635, "index_size": 236, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10710, "raw_average_key_size": 46, "raw_value_size": 79539, "raw_average_value_size": 348, "num_data_blocks": 11, "num_entries": 228, "num_filter_entries": 228, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726476793, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e2cb7dc7-eca3-4260-b4f6-b7f56eb2a9a3", "db_session_id": "VZLHX7RRL97P5E3G2IV4", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-16T08:53:13.397 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726476793104381, "job": 1, "event": "recovery_finished"} 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55730ac28e00 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: DB pointer 0x55730ad2a000 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout: ** DB Stats ** 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-16T08:53:13.398 INFO:journalctl@ceph.mon.a.smithi111.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: ** Compaction Stats [default] ** 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: L0 2/0 87.30 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: Sum 2/0 87.30 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: ** Compaction Stats [default] ** 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 156.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T08:53:13.399 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: Cumulative compaction: 0.00 GB write, 28.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: Interval compaction: 0.00 GB write, 28.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-16T08:53:13.400 INFO:journalctl@ceph.mon.a.smithi111.stdout: Block cache BinnedLRUCache@0x55730abf9610#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: starting mon.a rank 0 at public addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] at bind addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mon.a@-1(???) e1 preinit fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mon.a@-1(???).mds e1 new map 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mon.a@-1(???).mds e1 print_map 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: e1 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: btime 2024-09-16T08:53:08:312988+0000 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: legacy client fscid: -1 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2024-09-16T08:53:13.401 INFO:journalctl@ceph.mon.a.smithi111.stdout: No filesystems configured 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 bash[29766]: 676e47c984e2ea0d1182324f64bd7114a8677a1ebc2178b05fd635f520f22dd3 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 systemd[1]: Started Ceph mon.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: monmap epoch 1 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: last_changed 2024-09-16T08:53:05.472892+0000 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: created 2024-09-16T08:53:05.472892+0000 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: min_mon_release 19 (squid) 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: election_strategy: 1 2024-09-16T08:53:13.402 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: 0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2024-09-16T08:53:13.403 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: fsmap 2024-09-16T08:53:13.403 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: osdmap e1: 0 total, 0 up, 0 in 2024-09-16T08:53:13.403 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:13 smithi111 ceph-mon[29803]: mgrmap e1: no daemons active 2024-09-16T08:53:14.202 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.599+0000 7fb51d59f640 1 Processor -- start 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.599+0000 7fb51d59f640 1 -- start start 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.600+0000 7fb51d59f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb5181069f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.600+0000 7fb51d59f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb518106fe0 con 0x7fb5181065f0 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.600+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb5181069f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.600+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb5181069f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44996/0 (socket says 172.21.15.111:44996) 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.600+0000 7fb516ffd640 1 -- 172.21.15.111:0/1243622211 learned_addr learned my addr 172.21.15.111:0/1243622211 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.600+0000 7fb516ffd640 1 -- 172.21.15.111:0/1243622211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb518107160 con 0x7fb5181065f0 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.602+0000 7fb516ffd640 1 --2- 172.21.15.111:0/1243622211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb5181069f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb510009820 tx=0x7fb51002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a7fd9f615a57c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.602+0000 7fb515ffb640 1 -- 172.21.15.111:0/1243622211 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb510037070 con 0x7fb5181065f0 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.602+0000 7fb515ffb640 1 -- 172.21.15.111:0/1243622211 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fb510032870 con 0x7fb5181065f0 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.602+0000 7fb515ffb640 1 -- 172.21.15.111:0/1243622211 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5100305d0 con 0x7fb5181065f0 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.603+0000 7fb51d59f640 1 -- 172.21.15.111:0/1243622211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 msgr2=0x7fb5181069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.603+0000 7fb51d59f640 1 --2- 172.21.15.111:0/1243622211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb5181069f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb510009820 tx=0x7fb51002a270 comp rx=0 tx=0).stop 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.603+0000 7fb51d59f640 1 -- 172.21.15.111:0/1243622211 shutdown_connections 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.603+0000 7fb51d59f640 1 --2- 172.21.15.111:0/1243622211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb5181069f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.603+0000 7fb51d59f640 1 -- 172.21.15.111:0/1243622211 >> 172.21.15.111:0/1243622211 conn(0x7fb518101f20 msgr2=0x7fb518104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.603+0000 7fb51d59f640 1 -- 172.21.15.111:0/1243622211 shutdown_connections 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.603+0000 7fb51d59f640 1 -- 172.21.15.111:0/1243622211 wait complete. 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.603+0000 7fb51d59f640 1 Processor -- start 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb51d59f640 1 -- start start 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb51d59f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb51819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb51d59f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb51810aa70 con 0x7fb5181065f0 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb51819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb51819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45002/0 (socket says 172.21.15.111:45002) 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb516ffd640 1 -- 172.21.15.111:0/3454625607 learned_addr learned my addr 172.21.15.111:0/3454625607 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb516ffd640 1 -- 172.21.15.111:0/3454625607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51819abb0 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb516ffd640 1 --2- 172.21.15.111:0/3454625607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb51819a650 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb5100090b0 tx=0x7fb5100307c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb50ffff640 1 -- 172.21.15.111:0/3454625607 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb51003f020 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb50ffff640 1 -- 172.21.15.111:0/3454625607 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fb510030e20 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.604+0000 7fb51d59f640 1 -- 172.21.15.111:0/3454625607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51819ae40 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.605+0000 7fb50ffff640 1 -- 172.21.15.111:0/3454625607 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb510037040 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.605+0000 7fb51d59f640 1 -- 172.21.15.111:0/3454625607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51819b800 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.605+0000 7fb50ffff640 1 -- 172.21.15.111:0/3454625607 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb510042020 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.605+0000 7fb50ffff640 1 -- 172.21.15.111:0/3454625607 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb510044990 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.605+0000 7fb51d59f640 1 -- 172.21.15.111:0/3454625607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb51810abf0 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.608+0000 7fb50ffff640 1 -- 172.21.15.111:0/3454625607 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fb510049020 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.665+0000 7fb51d59f640 1 -- 172.21.15.111:0/3454625607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fb518106a70 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.667+0000 7fb50ffff640 1 -- 172.21.15.111:0/3454625607 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fb510044400 con 0x7fb5181065f0 2024-09-16T08:53:14.205 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.668+0000 7fb51d59f640 1 -- 172.21.15.111:0/3454625607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 msgr2=0x7fb51819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:14.206 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.668+0000 7fb51d59f640 1 --2- 172.21.15.111:0/3454625607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb51819a650 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb5100090b0 tx=0x7fb5100307c0 comp rx=0 tx=0).stop 2024-09-16T08:53:14.206 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.668+0000 7fb51d59f640 1 -- 172.21.15.111:0/3454625607 shutdown_connections 2024-09-16T08:53:14.206 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.668+0000 7fb51d59f640 1 --2- 172.21.15.111:0/3454625607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5181065f0 0x7fb51819a650 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:14.206 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.668+0000 7fb51d59f640 1 -- 172.21.15.111:0/3454625607 >> 172.21.15.111:0/3454625607 conn(0x7fb518101f20 msgr2=0x7fb518102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:14.206 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.668+0000 7fb51d59f640 1 -- 172.21.15.111:0/3454625607 shutdown_connections 2024-09-16T08:53:14.206 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:13.668+0000 7fb51d59f640 1 -- 172.21.15.111:0/3454625607 wait complete. 2024-09-16T08:53:14.206 INFO:teuthology.orchestra.run.smithi111.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-16T08:53:14.220 INFO:teuthology.orchestra.run.smithi111.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-16T08:53:14.220 INFO:teuthology.orchestra.run.smithi111.stdout:Creating mgr... 2024-09-16T08:53:14.220 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-16T08:53:14.221 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-16T08:53:14.462 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.y 2024-09-16T08:53:14.462 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-16T08:53:14.614 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f774c998-7408-11ef-bceb-c7b262605968.target.wants/ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-f774c998-7408-11ef-bceb-c7b262605968@.service. 2024-09-16T08:53:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3454625607' entity='client.admin' 2024-09-16T08:53:14.926 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:14 smithi111 podman[30039]: 2024-09-16 08:53:14.85604739 +0000 UTC m=+0.123078950 container create 5d7785ae384e4632538cd532d952d65838412b5b7fb6cbe112399a0851a4db5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T08:53:15.205 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T08:53:15.205 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2024-09-16T08:53:15.212 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T08:53:15.212 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-09-16T08:53:15.213 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2024-09-16T08:53:15.213 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T08:53:15.221 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T08:53:15.221 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2024-09-16T08:53:15.229 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T08:53:15.229 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-09-16T08:53:15.229 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2024-09-16T08:53:15.229 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-16T08:53:15.229 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr to start... 2024-09-16T08:53:15.229 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr... 2024-09-16T08:53:15.352 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:15 smithi111 podman[30039]: 2024-09-16 08:53:15.083489441 +0000 UTC m=+0.350521004 container init 5d7785ae384e4632538cd532d952d65838412b5b7fb6cbe112399a0851a4db5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-16T08:53:15.352 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:15 smithi111 podman[30039]: 2024-09-16 08:53:15.088211621 +0000 UTC m=+0.355243181 container start 5d7785ae384e4632538cd532d952d65838412b5b7fb6cbe112399a0851a4db5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-16T08:53:15.352 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:15 smithi111 bash[30039]: 5d7785ae384e4632538cd532d952d65838412b5b7fb6cbe112399a0851a4db5b 2024-09-16T08:53:15.352 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:15 smithi111 systemd[1]: Started Ceph mgr.y for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:53:15.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2973003090' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-16T08:53:16.326 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:16.340 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-09-16T08:53:16.340 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsid": "f774c998-7408-11ef-bceb-c7b262605968", 2024-09-16T08:53:16.340 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T08:53:16.340 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T08:53:16.340 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T08:53:16.340 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T08:53:16.340 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 0 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T08:53:16.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T08:53:08:312988+0000", 2024-09-16T08:53:16.343 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T08:53:16.344 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T08:53:08.313676+0000", 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.676+0000 7fc14e059640 1 Processor -- start 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.676+0000 7fc14e059640 1 -- start start 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.676+0000 7fc14e059640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc148109180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.676+0000 7fc14e059640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481096c0 con 0x7fc148079bc0 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.677+0000 7fc1477fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc148109180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.677+0000 7fc1477fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc148109180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45028/0 (socket says 172.21.15.111:45028) 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.677+0000 7fc1477fe640 1 -- 172.21.15.111:0/3174803398 learned_addr learned my addr 172.21.15.111:0/3174803398 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.677+0000 7fc1477fe640 1 -- 172.21.15.111:0/3174803398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc148109840 con 0x7fc148079bc0 2024-09-16T08:53:16.346 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc1477fe640 1 --2- 172.21.15.111:0/3174803398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc148109180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc138009820 tx=0x7fc13802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78dd8a19cdb6fdbd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:16.346 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc1467fc640 1 -- 172.21.15.111:0/3174803398 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc138037070 con 0x7fc148079bc0 2024-09-16T08:53:16.346 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc1467fc640 1 -- 172.21.15.111:0/3174803398 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fc138032870 con 0x7fc148079bc0 2024-09-16T08:53:16.346 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc1467fc640 1 -- 172.21.15.111:0/3174803398 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc138030660 con 0x7fc148079bc0 2024-09-16T08:53:16.346 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc14e059640 1 -- 172.21.15.111:0/3174803398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 msgr2=0x7fc148109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:16.346 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc14e059640 1 --2- 172.21.15.111:0/3174803398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc148109180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc138009820 tx=0x7fc13802a300 comp rx=0 tx=0).stop 2024-09-16T08:53:16.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc14e059640 1 -- 172.21.15.111:0/3174803398 shutdown_connections 2024-09-16T08:53:16.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc14e059640 1 --2- 172.21.15.111:0/3174803398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc148109180 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:16.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc14e059640 1 -- 172.21.15.111:0/3174803398 >> 172.21.15.111:0/3174803398 conn(0x7fc148100df0 msgr2=0x7fc148103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:16.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc14e059640 1 -- 172.21.15.111:0/3174803398 shutdown_connections 2024-09-16T08:53:16.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.678+0000 7fc14e059640 1 -- 172.21.15.111:0/3174803398 wait complete. 2024-09-16T08:53:16.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.679+0000 7fc14e059640 1 Processor -- start 2024-09-16T08:53:16.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.679+0000 7fc14e059640 1 -- start start 2024-09-16T08:53:16.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.679+0000 7fc14e059640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc14819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:16.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.679+0000 7fc14e059640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc14810a8b0 con 0x7fc148079bc0 2024-09-16T08:53:16.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.679+0000 7fc1477fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc14819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:16.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.679+0000 7fc1477fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc14819a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45036/0 (socket says 172.21.15.111:45036) 2024-09-16T08:53:16.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.679+0000 7fc1477fe640 1 -- 172.21.15.111:0/2973003090 learned_addr learned my addr 172.21.15.111:0/2973003090 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:16.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.680+0000 7fc1477fe640 1 -- 172.21.15.111:0/2973003090 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14819abf0 con 0x7fc148079bc0 2024-09-16T08:53:16.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.680+0000 7fc1477fe640 1 --2- 172.21.15.111:0/2973003090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc14819a690 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc138030380 tx=0x7fc138030b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:16.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.680+0000 7fc144ff9640 1 -- 172.21.15.111:0/2973003090 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc138037040 con 0x7fc148079bc0 2024-09-16T08:53:16.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.680+0000 7fc144ff9640 1 -- 172.21.15.111:0/2973003090 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fc138040070 con 0x7fc148079bc0 2024-09-16T08:53:16.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.680+0000 7fc144ff9640 1 -- 172.21.15.111:0/2973003090 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc13803a390 con 0x7fc148079bc0 2024-09-16T08:53:16.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.680+0000 7fc14e059640 1 -- 172.21.15.111:0/2973003090 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14819ae80 con 0x7fc148079bc0 2024-09-16T08:53:16.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.680+0000 7fc14e059640 1 -- 172.21.15.111:0/2973003090 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14819b840 con 0x7fc148079bc0 2024-09-16T08:53:16.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.680+0000 7fc144ff9640 1 -- 172.21.15.111:0/2973003090 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc13803a690 con 0x7fc148079bc0 2024-09-16T08:53:16.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.680+0000 7fc144ff9640 1 -- 172.21.15.111:0/2973003090 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc1380458c0 con 0x7fc148079bc0 2024-09-16T08:53:16.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.681+0000 7fc14e059640 1 -- 172.21.15.111:0/2973003090 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1481071f0 con 0x7fc148079bc0 2024-09-16T08:53:16.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.683+0000 7fc144ff9640 1 -- 172.21.15.111:0/2973003090 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fc138045460 con 0x7fc148079bc0 2024-09-16T08:53:16.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.745+0000 7fc14e059640 1 -- 172.21.15.111:0/2973003090 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc14819bdf0 con 0x7fc148079bc0 2024-09-16T08:53:16.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.746+0000 7fc144ff9640 1 -- 172.21.15.111:0/2973003090 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7fc138045b10 con 0x7fc148079bc0 2024-09-16T08:53:16.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.747+0000 7fc14e059640 1 -- 172.21.15.111:0/2973003090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 msgr2=0x7fc14819a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:16.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.747+0000 7fc14e059640 1 --2- 172.21.15.111:0/2973003090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc14819a690 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc138030380 tx=0x7fc138030b40 comp rx=0 tx=0).stop 2024-09-16T08:53:16.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.747+0000 7fc14e059640 1 -- 172.21.15.111:0/2973003090 shutdown_connections 2024-09-16T08:53:16.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.747+0000 7fc14e059640 1 --2- 172.21.15.111:0/2973003090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc148079bc0 0x7fc14819a690 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:16.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.747+0000 7fc14e059640 1 -- 172.21.15.111:0/2973003090 >> 172.21.15.111:0/2973003090 conn(0x7fc148100df0 msgr2=0x7fc148101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:16.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.747+0000 7fc14e059640 1 -- 172.21.15.111:0/2973003090 shutdown_connections 2024-09-16T08:53:16.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:15.747+0000 7fc14e059640 1 -- 172.21.15.111:0/2973003090 wait complete. 2024-09-16T08:53:16.352 INFO:teuthology.orchestra.run.smithi111.stdout:mgr not available, waiting (1/15)... 2024-09-16T08:53:19.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1339476768' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-16T08:53:19.504 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:19.504 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-09-16T08:53:19.504 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsid": "f774c998-7408-11ef-bceb-c7b262605968", 2024-09-16T08:53:19.504 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 0 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-16T08:53:19.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T08:53:08:312988+0000", 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T08:53:19.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T08:53:19.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:19.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T08:53:19.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T08:53:19.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T08:53:19.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T08:53:19.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T08:53:19.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T08:53:19.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T08:53:19.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T08:53:08.313676+0000", 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.798+0000 7f0dd491e640 1 Processor -- start 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.798+0000 7f0dd491e640 1 -- start start 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.798+0000 7f0dd491e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:19.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.799+0000 7f0dd491e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd0105eb0 con 0x7f0dd01054c0 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.799+0000 7f0dce59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.799+0000 7f0dce59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01058c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45042/0 (socket says 172.21.15.111:45042) 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.799+0000 7f0dce59c640 1 -- 172.21.15.111:0/3086565693 learned_addr learned my addr 172.21.15.111:0/3086565693 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.799+0000 7f0dce59c640 1 -- 172.21.15.111:0/3086565693 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd0106030 con 0x7f0dd01054c0 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.800+0000 7f0dce59c640 1 --2- 172.21.15.111:0/3086565693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01058c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009820 tx=0x7f0dc002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e044b4407c1c92ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.800+0000 7f0dcd59a640 1 -- 172.21.15.111:0/3086565693 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dc0037070 con 0x7f0dd01054c0 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.800+0000 7f0dcd59a640 1 -- 172.21.15.111:0/3086565693 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f0dc0032870 con 0x7f0dd01054c0 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.800+0000 7f0dcd59a640 1 -- 172.21.15.111:0/3086565693 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dc00305d0 con 0x7f0dd01054c0 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.800+0000 7f0dd491e640 1 -- 172.21.15.111:0/3086565693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 msgr2=0x7f0dd01058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.800+0000 7f0dd491e640 1 --2- 172.21.15.111:0/3086565693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01058c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009820 tx=0x7f0dc002a270 comp rx=0 tx=0).stop 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.801+0000 7f0dd491e640 1 -- 172.21.15.111:0/3086565693 shutdown_connections 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.801+0000 7f0dd491e640 1 --2- 172.21.15.111:0/3086565693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01058c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:19.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.801+0000 7f0dd491e640 1 -- 172.21.15.111:0/3086565693 >> 172.21.15.111:0/3086565693 conn(0x7f0dd0100df0 msgr2=0x7f0dd0103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.801+0000 7f0dd491e640 1 -- 172.21.15.111:0/3086565693 shutdown_connections 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.801+0000 7f0dd491e640 1 -- 172.21.15.111:0/3086565693 wait complete. 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.801+0000 7f0dd491e640 1 Processor -- start 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.801+0000 7f0dd491e640 1 -- start start 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dd491e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01903e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dd491e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd0109940 con 0x7f0dd01054c0 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dce59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01903e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dce59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01903e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45056/0 (socket says 172.21.15.111:45056) 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dce59c640 1 -- 172.21.15.111:0/1339476768 learned_addr learned my addr 172.21.15.111:0/1339476768 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dce59c640 1 -- 172.21.15.111:0/1339476768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd0190940 con 0x7f0dd01054c0 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dce59c640 1 --2- 172.21.15.111:0/1339476768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01903e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00097f0 tx=0x7f0dc0030b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dbf7fe640 1 -- 172.21.15.111:0/1339476768 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dc003f020 con 0x7f0dd01054c0 2024-09-16T08:53:19.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dbf7fe640 1 -- 172.21.15.111:0/1339476768 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f0dc0030dc0 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dd491e640 1 -- 172.21.15.111:0/1339476768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd018ed30 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dbf7fe640 1 -- 172.21.15.111:0/1339476768 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dc0037040 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.802+0000 7f0dd491e640 1 -- 172.21.15.111:0/1339476768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd018f2b0 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.803+0000 7f0dbf7fe640 1 -- 172.21.15.111:0/1339476768 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0dc0042020 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.803+0000 7f0dbf7fe640 1 -- 172.21.15.111:0/1339476768 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0dc0044980 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.803+0000 7f0dd491e640 1 -- 172.21.15.111:0/1339476768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd0109ac0 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.805+0000 7f0dbf7fe640 1 -- 172.21.15.111:0/1339476768 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0dc0049020 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.868+0000 7f0dd491e640 1 -- 172.21.15.111:0/1339476768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0dd018f790 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.869+0000 7f0dbf7fe640 1 -- 172.21.15.111:0/1339476768 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f0dc0044400 con 0x7f0dd01054c0 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.870+0000 7f0dd491e640 1 -- 172.21.15.111:0/1339476768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 msgr2=0x7f0dd01903e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.870+0000 7f0dd491e640 1 --2- 172.21.15.111:0/1339476768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01903e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00097f0 tx=0x7f0dc0030b00 comp rx=0 tx=0).stop 2024-09-16T08:53:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.870+0000 7f0dd491e640 1 -- 172.21.15.111:0/1339476768 shutdown_connections 2024-09-16T08:53:19.513 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.870+0000 7f0dd491e640 1 --2- 172.21.15.111:0/1339476768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dd01054c0 0x7f0dd01903e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:19.513 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.870+0000 7f0dd491e640 1 -- 172.21.15.111:0/1339476768 >> 172.21.15.111:0/1339476768 conn(0x7f0dd0100df0 msgr2=0x7f0dd0078f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:19.513 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.870+0000 7f0dd491e640 1 -- 172.21.15.111:0/1339476768 shutdown_connections 2024-09-16T08:53:19.513 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:18.870+0000 7f0dd491e640 1 -- 172.21.15.111:0/1339476768 wait complete. 2024-09-16T08:53:19.513 INFO:teuthology.orchestra.run.smithi111.stdout:mgr not available, waiting (2/15)... 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: Activating manager daemon y 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: mgrmap e2: y(active, starting, since 0.00325597s) 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: Manager daemon y is now available 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T08:53:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T08:53:20.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' 2024-09-16T08:53:20.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' 2024-09-16T08:53:20.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:20 smithi111 ceph-mon[29803]: from='mgr.14100 172.21.15.111:0/111018572' entity='mgr.y' 2024-09-16T08:53:22.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:22 smithi111 ceph-mon[29803]: mgrmap e3: y(active, since 1.00731s) 2024-09-16T08:53:23.425 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:23 smithi111 ceph-mon[29803]: mgrmap e4: y(active, since 2s) 2024-09-16T08:53:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2573129094' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-16T08:53:31.794 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:31.794 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-09-16T08:53:31.794 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsid": "f774c998-7408-11ef-bceb-c7b262605968", 2024-09-16T08:53:31.794 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 0 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-09-16T08:53:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T08:53:31.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T08:53:08:312988+0000", 2024-09-16T08:53:31.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T08:53:31.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T08:53:08.313676+0000", 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.151+0000 7f011cc3f640 1 Processor -- start 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.151+0000 7f011cc3f640 1 -- start start 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.151+0000 7f011cc3f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f0118106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.151+0000 7f011cc3f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0118107000 con 0x7f0118106610 2024-09-16T08:53:31.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.151+0000 7f011659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f0118106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.151+0000 7f011659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f0118106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57444/0 (socket says 172.21.15.111:57444) 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.151+0000 7f011659c640 1 -- 172.21.15.111:0/2861387774 learned_addr learned my addr 172.21.15.111:0/2861387774 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.152+0000 7f011659c640 1 -- 172.21.15.111:0/2861387774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0118107180 con 0x7f0118106610 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.152+0000 7f011659c640 1 --2- 172.21.15.111:0/2861387774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f0118106a10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0108009820 tx=0x7f010802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322ab9385d222a97 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011559a640 1 -- 172.21.15.111:0/2861387774 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0108037070 con 0x7f0118106610 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011559a640 1 -- 172.21.15.111:0/2861387774 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f0108032870 con 0x7f0118106610 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011559a640 1 -- 172.21.15.111:0/2861387774 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01080305d0 con 0x7f0118106610 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011cc3f640 1 -- 172.21.15.111:0/2861387774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 msgr2=0x7f0118106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011cc3f640 1 --2- 172.21.15.111:0/2861387774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f0118106a10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0108009820 tx=0x7f010802a270 comp rx=0 tx=0).stop 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011cc3f640 1 -- 172.21.15.111:0/2861387774 shutdown_connections 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011cc3f640 1 --2- 172.21.15.111:0/2861387774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f0118106a10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011cc3f640 1 -- 172.21.15.111:0/2861387774 >> 172.21.15.111:0/2861387774 conn(0x7f0118101f00 msgr2=0x7f0118104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:31.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011cc3f640 1 -- 172.21.15.111:0/2861387774 shutdown_connections 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.153+0000 7f011cc3f640 1 -- 172.21.15.111:0/2861387774 wait complete. 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.154+0000 7f011cc3f640 1 Processor -- start 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.154+0000 7f011cc3f640 1 -- start start 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.154+0000 7f011cc3f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f011807c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.154+0000 7f011cc3f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f011810aa90 con 0x7f0118106610 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.154+0000 7f011659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f011807c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.154+0000 7f011659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f011807c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57460/0 (socket says 172.21.15.111:57460) 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.154+0000 7f011659c640 1 -- 172.21.15.111:0/2573129094 learned_addr learned my addr 172.21.15.111:0/2573129094 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.155+0000 7f011659c640 1 -- 172.21.15.111:0/2573129094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011807c6e0 con 0x7f0118106610 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.155+0000 7f011659c640 1 --2- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f011807c180 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f01080090b0 tx=0x7f01080307c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.155+0000 7f01077fe640 1 -- 172.21.15.111:0/2573129094 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f010803f020 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.155+0000 7f01077fe640 1 -- 172.21.15.111:0/2573129094 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f0108030e20 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.155+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011807aad0 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.155+0000 7f01077fe640 1 -- 172.21.15.111:0/2573129094 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0108037040 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.155+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f011807b050 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.156+0000 7f01077fe640 1 -- 172.21.15.111:0/2573129094 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) ==== 51754+0+0 (secure 0 0 0) 0x7f0108042020 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.156+0000 7f01077fe640 1 --2- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f010003f600 0x7f0100041ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.156+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f011810ac10 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.156+0000 7f01077fe640 1 -- 172.21.15.111:0/2573129094 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0108074370 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.156+0000 7f0115d9b640 1 --2- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f010003f600 0x7f0100041ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.157+0000 7f0115d9b640 1 --2- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f010003f600 0x7f0100041ac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f010c0098c0 tx=0x7f010c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.161+0000 7f01077fe640 1 -- 172.21.15.111:0/2573129094 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f0108049020 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.406+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f011807b590 con 0x7f0118106610 2024-09-16T08:53:31.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.407+0000 7f01077fe640 1 -- 172.21.15.111:0/2573129094 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f01080740d0 con 0x7f0118106610 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f010003f600 msgr2=0x7f0100041ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 --2- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f010003f600 0x7f0100041ac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f010c0098c0 tx=0x7f010c006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 msgr2=0x7f011807c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 --2- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f011807c180 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f01080090b0 tx=0x7f01080307c0 comp rx=0 tx=0).stop 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 shutdown_connections 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 --2- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f010003f600 0x7f0100041ac0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 --2- 172.21.15.111:0/2573129094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0118106610 0x7f011807c180 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 >> 172.21.15.111:0/2573129094 conn(0x7f0118101f00 msgr2=0x7f0118102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 shutdown_connections 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:25.409+0000 7f011cc3f640 1 -- 172.21.15.111:0/2573129094 wait complete. 2024-09-16T08:53:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:mgr is available 2024-09-16T08:53:33.786 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1909720352' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-16T08:53:34.087 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout fsid = f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-09-16T08:53:34.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.289+0000 7f31ff7ea640 1 Processor -- start 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.289+0000 7f31ff7ea640 1 -- start start 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.289+0000 7f31ff7ea640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f8106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.289+0000 7f31ff7ea640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f31f8107000 con 0x7f31f8106610 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.289+0000 7f31fd586640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f8106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.289+0000 7f31fd586640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f8106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43950/0 (socket says 172.21.15.111:43950) 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.289+0000 7f31fd586640 1 -- 172.21.15.111:0/1409009498 learned_addr learned my addr 172.21.15.111:0/1409009498 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.290+0000 7f31fd586640 1 -- 172.21.15.111:0/1409009498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f8107180 con 0x7f31f8106610 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.290+0000 7f31fd586640 1 --2- 172.21.15.111:0/1409009498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f8106a10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009820 tx=0x7f31f002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9be326e64cce157 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.290+0000 7f31f7fff640 1 -- 172.21.15.111:0/1409009498 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31f0037070 con 0x7f31f8106610 2024-09-16T08:53:34.089 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.290+0000 7f31f7fff640 1 -- 172.21.15.111:0/1409009498 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f31f0032870 con 0x7f31f8106610 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.291+0000 7f31f7fff640 1 -- 172.21.15.111:0/1409009498 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31f00305d0 con 0x7f31f8106610 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.291+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1409009498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 msgr2=0x7f31f8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.291+0000 7f31ff7ea640 1 --2- 172.21.15.111:0/1409009498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f8106a10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009820 tx=0x7f31f002a270 comp rx=0 tx=0).stop 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.291+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1409009498 shutdown_connections 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.291+0000 7f31ff7ea640 1 --2- 172.21.15.111:0/1409009498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f8106a10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.291+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1409009498 >> 172.21.15.111:0/1409009498 conn(0x7f31f8101f00 msgr2=0x7f31f8104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.291+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1409009498 shutdown_connections 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.291+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1409009498 wait complete. 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31ff7ea640 1 Processor -- start 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31ff7ea640 1 -- start start 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31ff7ea640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31ff7ea640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f31f810aa90 con 0x7f31f8106610 2024-09-16T08:53:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31fd586640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31fd586640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43958/0 (socket says 172.21.15.111:43958) 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31fd586640 1 -- 172.21.15.111:0/1909720352 learned_addr learned my addr 172.21.15.111:0/1909720352 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31fd586640 1 -- 172.21.15.111:0/1909720352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f819abb0 con 0x7f31f8106610 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31fd586640 1 --2- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f819a650 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31f00097f0 tx=0x7f31f0030b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.292+0000 7f31f67fc640 1 -- 172.21.15.111:0/1909720352 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31f003f020 con 0x7f31f8106610 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.293+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f819ae40 con 0x7f31f8106610 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.293+0000 7f31f67fc640 1 -- 172.21.15.111:0/1909720352 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f31f0030e40 con 0x7f31f8106610 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.293+0000 7f31f67fc640 1 -- 172.21.15.111:0/1909720352 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31f0037040 con 0x7f31f8106610 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.293+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f819b800 con 0x7f31f8106610 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.294+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f810ac10 con 0x7f31f8106610 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.294+0000 7f31f67fc640 1 -- 172.21.15.111:0/1909720352 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) ==== 51754+0+0 (secure 0 0 0) 0x7f31f0042020 con 0x7f31f8106610 2024-09-16T08:53:34.091 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.294+0000 7f31f67fc640 1 --2- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f31e003f600 0x7f31e0041ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.294+0000 7f31f67fc640 1 -- 172.21.15.111:0/1909720352 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f31f00741f0 con 0x7f31f8106610 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.294+0000 7f31fcd85640 1 --2- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f31e003f600 0x7f31e0041ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.294+0000 7f31fcd85640 1 --2- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f31e003f600 0x7f31e0041ac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0098c0 tx=0x7f31ec006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.299+0000 7f31f67fc640 1 -- 172.21.15.111:0/1909720352 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f31f0049020 con 0x7f31f8106610 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.476+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f31f819bfd0 con 0x7f31f8106610 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.477+0000 7f31f67fc640 1 -- 172.21.15.111:0/1909720352 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+376 (secure 0 0 0) 0x7f31f003f1c0 con 0x7f31f8106610 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f31e003f600 msgr2=0x7f31e0041ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 --2- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f31e003f600 0x7f31e0041ac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0098c0 tx=0x7f31ec006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 msgr2=0x7f31f819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 --2- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f819a650 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31f00097f0 tx=0x7f31f0030b00 comp rx=0 tx=0).stop 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 shutdown_connections 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 --2- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f31e003f600 0x7f31e0041ac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 --2- 172.21.15.111:0/1909720352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f8106610 0x7f31f819a650 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:34.092 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 >> 172.21.15.111:0/1909720352 conn(0x7f31f8101f00 msgr2=0x7f31f8102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:34.093 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 shutdown_connections 2024-09-16T08:53:34.093 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:33.479+0000 7f31ff7ea640 1 -- 172.21.15.111:0/1909720352 wait complete. 2024-09-16T08:53:34.093 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling cephadm module... 2024-09-16T08:53:35.094 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1879431811' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-16T08:53:35.095 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:34 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ignoring --setuser ceph since I am not root 2024-09-16T08:53:35.095 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:34 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ignoring --setgroup ceph since I am not root 2024-09-16T08:53:35.388 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.477+0000 7feee7e7f640 1 Processor -- start 2024-09-16T08:53:35.388 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.477+0000 7feee7e7f640 1 -- start start 2024-09-16T08:53:35.388 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.477+0000 7feee7e7f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee0105090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:35.388 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.477+0000 7feee7e7f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feee0105680 con 0x7feee0104c90 2024-09-16T08:53:35.389 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.477+0000 7feee5c1b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee0105090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:35.389 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.477+0000 7feee5c1b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee0105090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43962/0 (socket says 172.21.15.111:43962) 2024-09-16T08:53:35.389 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.477+0000 7feee5c1b640 1 -- 172.21.15.111:0/1083557813 learned_addr learned my addr 172.21.15.111:0/1083557813 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:35.389 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.478+0000 7feee5c1b640 1 -- 172.21.15.111:0/1083557813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee0105800 con 0x7feee0104c90 2024-09-16T08:53:35.389 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.478+0000 7feee5c1b640 1 --2- 172.21.15.111:0/1083557813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee0105090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feedc00a9c0 tx=0x7feedc02b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a141b38fbbe30894 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:35.389 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.478+0000 7feee4c19640 1 -- 172.21.15.111:0/1083557813 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feedc00b040 con 0x7feee0104c90 2024-09-16T08:53:35.389 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.478+0000 7feee4c19640 1 -- 172.21.15.111:0/1083557813 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7feedc030a70 con 0x7feee0104c90 2024-09-16T08:53:35.390 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.478+0000 7feee4c19640 1 -- 172.21.15.111:0/1083557813 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feedc0329c0 con 0x7feee0104c90 2024-09-16T08:53:35.390 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.479+0000 7feee7e7f640 1 -- 172.21.15.111:0/1083557813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 msgr2=0x7feee0105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:35.390 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.479+0000 7feee7e7f640 1 --2- 172.21.15.111:0/1083557813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee0105090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feedc00a9c0 tx=0x7feedc02b610 comp rx=0 tx=0).stop 2024-09-16T08:53:35.390 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.479+0000 7feee7e7f640 1 -- 172.21.15.111:0/1083557813 shutdown_connections 2024-09-16T08:53:35.390 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.479+0000 7feee7e7f640 1 --2- 172.21.15.111:0/1083557813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee0105090 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:35.390 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.479+0000 7feee7e7f640 1 -- 172.21.15.111:0/1083557813 >> 172.21.15.111:0/1083557813 conn(0x7feee0100560 msgr2=0x7feee01029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:35.390 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.479+0000 7feee7e7f640 1 -- 172.21.15.111:0/1083557813 shutdown_connections 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.479+0000 7feee7e7f640 1 -- 172.21.15.111:0/1083557813 wait complete. 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.479+0000 7feee7e7f640 1 Processor -- start 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feee7e7f640 1 -- start start 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feee7e7f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feee7e7f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feee0106430 con 0x7feee0104c90 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feee5c1b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feee5c1b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee019a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43964/0 (socket says 172.21.15.111:43964) 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feee5c1b640 1 -- 172.21.15.111:0/1879431811 learned_addr learned my addr 172.21.15.111:0/1879431811 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feee5c1b640 1 -- 172.21.15.111:0/1879431811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee019ac30 con 0x7feee0104c90 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feee5c1b640 1 --2- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee019a6d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feedc02bbd0 tx=0x7feedc0315d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feed2ffd640 1 -- 172.21.15.111:0/1879431811 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feedc040070 con 0x7feee0104c90 2024-09-16T08:53:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feed2ffd640 1 -- 172.21.15.111:0/1879431811 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7feedc031d20 con 0x7feee0104c90 2024-09-16T08:53:35.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee019aec0 con 0x7feee0104c90 2024-09-16T08:53:35.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.480+0000 7feed2ffd640 1 -- 172.21.15.111:0/1879431811 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feedc03bb10 con 0x7feee0104c90 2024-09-16T08:53:35.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.481+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee019b880 con 0x7feee0104c90 2024-09-16T08:53:35.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.481+0000 7feed2ffd640 1 -- 172.21.15.111:0/1879431811 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) ==== 51754+0+0 (secure 0 0 0) 0x7feedc043070 con 0x7feee0104c90 2024-09-16T08:53:35.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.481+0000 7feed2ffd640 1 --2- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7feecc03f650 0x7feecc041b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:35.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.482+0000 7feed2ffd640 1 -- 172.21.15.111:0/1879431811 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feedc074e50 con 0x7feee0104c90 2024-09-16T08:53:35.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.482+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee0106af0 con 0x7feee0104c90 2024-09-16T08:53:35.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.482+0000 7feee541a640 1 --2- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7feecc03f650 0x7feecc041b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:35.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.482+0000 7feee541a640 1 --2- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7feecc03f650 0x7feecc041b10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feed40098c0 tx=0x7feed4006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:35.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.486+0000 7feed2ffd640 1 -- 172.21.15.111:0/1879431811 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7feedc074870 con 0x7feee0104c90 2024-09-16T08:53:35.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.709+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7feee019c050 con 0x7feee0104c90 2024-09-16T08:53:35.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.761+0000 7feed2ffd640 1 -- 172.21.15.111:0/1879431811 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7feedc03ae10 con 0x7feee0104c90 2024-09-16T08:53:35.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.763+0000 7feed2ffd640 1 -- 172.21.15.111:0/1879431811 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 5) ==== 51765+0+0 (secure 0 0 0) 0x7feedc03b430 con 0x7feee0104c90 2024-09-16T08:53:35.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.763+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7feecc03f650 msgr2=0x7feecc041b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:35.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.763+0000 7feee7e7f640 1 --2- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7feecc03f650 0x7feecc041b10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feed40098c0 tx=0x7feed4006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:35.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.764+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 msgr2=0x7feee019a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:35.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.764+0000 7feee7e7f640 1 --2- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee019a6d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feedc02bbd0 tx=0x7feedc0315d0 comp rx=0 tx=0).stop 2024-09-16T08:53:35.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.764+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 shutdown_connections 2024-09-16T08:53:35.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.764+0000 7feee7e7f640 1 --2- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7feecc03f650 0x7feecc041b10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:35.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.764+0000 7feee7e7f640 1 --2- 172.21.15.111:0/1879431811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee0104c90 0x7feee019a6d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:35.395 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.764+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 >> 172.21.15.111:0/1879431811 conn(0x7feee0100560 msgr2=0x7feee018f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:35.395 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.764+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 shutdown_connections 2024-09-16T08:53:35.395 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:34.764+0000 7feee7e7f640 1 -- 172.21.15.111:0/1879431811 wait complete. 2024-09-16T08:53:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1879431811' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-16T08:53:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:35 smithi111 ceph-mon[29803]: mgrmap e5: y(active, since 14s) 2024-09-16T08:53:36.726 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-09-16T08:53:36.726 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-16T08:53:36.726 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T08:53:36.726 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-16T08:53:36.726 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T08:53:36.726 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-09-16T08:53:36.726 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.849+0000 7f72b1192640 1 Processor -- start 2024-09-16T08:53:36.726 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.849+0000 7f72b1192640 1 -- start start 2024-09-16T08:53:36.727 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.849+0000 7f72b1192640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac1058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:36.727 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.849+0000 7f72b1192640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac105eb0 con 0x7f72ac1054c0 2024-09-16T08:53:36.727 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.849+0000 7f72aad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac1058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:36.727 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.849+0000 7f72aad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac1058c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43978/0 (socket says 172.21.15.111:43978) 2024-09-16T08:53:36.727 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.849+0000 7f72aad9d640 1 -- 172.21.15.111:0/1489246057 learned_addr learned my addr 172.21.15.111:0/1489246057 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:36.727 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.850+0000 7f72aad9d640 1 -- 172.21.15.111:0/1489246057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ac106030 con 0x7f72ac1054c0 2024-09-16T08:53:36.727 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.850+0000 7f72aad9d640 1 --2- 172.21.15.111:0/1489246057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac1058c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f729c009820 tx=0x7f729c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=371d5fcf76859c50 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:36.727 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.850+0000 7f72a9d9b640 1 -- 172.21.15.111:0/1489246057 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f729c037070 con 0x7f72ac1054c0 2024-09-16T08:53:36.727 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.850+0000 7f72a9d9b640 1 -- 172.21.15.111:0/1489246057 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f729c032870 con 0x7f72ac1054c0 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.850+0000 7f72a9d9b640 1 -- 172.21.15.111:0/1489246057 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f729c0305d0 con 0x7f72ac1054c0 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.851+0000 7f72b1192640 1 -- 172.21.15.111:0/1489246057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 msgr2=0x7f72ac1058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.851+0000 7f72b1192640 1 --2- 172.21.15.111:0/1489246057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac1058c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f729c009820 tx=0x7f729c02a270 comp rx=0 tx=0).stop 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.851+0000 7f72b1192640 1 -- 172.21.15.111:0/1489246057 shutdown_connections 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.851+0000 7f72b1192640 1 --2- 172.21.15.111:0/1489246057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac1058c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.851+0000 7f72b1192640 1 -- 172.21.15.111:0/1489246057 >> 172.21.15.111:0/1489246057 conn(0x7f72ac100df0 msgr2=0x7f72ac103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.851+0000 7f72b1192640 1 -- 172.21.15.111:0/1489246057 shutdown_connections 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.851+0000 7f72b1192640 1 -- 172.21.15.111:0/1489246057 wait complete. 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.852+0000 7f72b1192640 1 Processor -- start 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.852+0000 7f72b1192640 1 -- start start 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.852+0000 7f72b1192640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:36.728 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.852+0000 7f72b1192640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac109940 con 0x7f72ac1054c0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.852+0000 7f72aad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.852+0000 7f72aad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac19a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43994/0 (socket says 172.21.15.111:43994) 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.852+0000 7f72aad9d640 1 -- 172.21.15.111:0/463764514 learned_addr learned my addr 172.21.15.111:0/463764514 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.852+0000 7f72aad9d640 1 -- 172.21.15.111:0/463764514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ac19ac20 con 0x7f72ac1054c0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.853+0000 7f72aad9d640 1 --2- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac19a6c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f729c0097f0 tx=0x7f729c030b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.853+0000 7f72a3fff640 1 -- 172.21.15.111:0/463764514 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f729c03f020 con 0x7f72ac1054c0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.853+0000 7f72a3fff640 1 -- 172.21.15.111:0/463764514 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f729c030dc0 con 0x7f72ac1054c0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.853+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72ac19aeb0 con 0x7f72ac1054c0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.853+0000 7f72a3fff640 1 -- 172.21.15.111:0/463764514 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f729c037040 con 0x7f72ac1054c0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.853+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72ac19b870 con 0x7f72ac1054c0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.854+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72ac109ac0 con 0x7f72ac1054c0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.854+0000 7f72a3fff640 1 -- 172.21.15.111:0/463764514 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) ==== 51765+0+0 (secure 0 0 0) 0x7f729c042020 con 0x7f72ac1054c0 2024-09-16T08:53:36.729 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.854+0000 7f72a3fff640 1 --2- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f729803f650 0x7f7298041b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.854+0000 7f72a3fff640 1 -- 172.21.15.111:0/463764514 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f729c0741f0 con 0x7f72ac1054c0 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.854+0000 7f72aa59c640 1 -- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f729803f650 msgr2=0x7f7298041b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2026406736 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.854+0000 7f72aa59c640 1 --2- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f729803f650 0x7f7298041b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:35.863+0000 7f72a3fff640 1 -- 172.21.15.111:0/463764514 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f729c049020 con 0x7f72ac1054c0 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.056+0000 7f72aa59c640 1 -- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f729803f650 msgr2=0x7f7298041b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2026406736 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.056+0000 7f72aa59c640 1 --2- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f729803f650 0x7f7298041b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.096+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f72ac19bf20 con 0x7f72ac1054c0 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.097+0000 7f72a3fff640 1 -- 172.21.15.111:0/463764514 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f729c03ae60 con 0x7f72ac1054c0 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f729803f650 msgr2=0x7f7298041b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 --2- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f729803f650 0x7f7298041b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 msgr2=0x7f72ac19a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 --2- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac19a6c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f729c0097f0 tx=0x7f729c030b00 comp rx=0 tx=0).stop 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 shutdown_connections 2024-09-16T08:53:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 --2- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f729803f650 0x7f7298041b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:36.731 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 --2- 172.21.15.111:0/463764514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72ac1054c0 0x7f72ac19a6c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:36.731 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 >> 172.21.15.111:0/463764514 conn(0x7f72ac100df0 msgr2=0x7f72ac078f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:36.731 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 shutdown_connections 2024-09-16T08:53:36.731 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:36.099+0000 7f72b1192640 1 -- 172.21.15.111:0/463764514 wait complete. 2024-09-16T08:53:36.731 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for the mgr to restart... 2024-09-16T08:53:36.731 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr epoch 5... 2024-09-16T08:53:37.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/463764514' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-16T08:53:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: Active manager daemon y restarted 2024-09-16T08:53:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: Activating manager daemon y 2024-09-16T08:53:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: osdmap e2: 0 total, 0 up, 0 in 2024-09-16T08:53:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: mgrmap e6: y(active, starting, since 0.00368622s) 2024-09-16T08:53:40.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:53:40.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T08:53:40.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T08:53:40.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T08:53:40.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T08:53:40.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:39 smithi111 ceph-mon[29803]: Manager daemon y is now available 2024-09-16T08:53:41.344 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:41.344 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:41.344 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: Found migration_current of "None". Setting to last migration. 2024-09-16T08:53:41.344 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:41.344 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:41.344 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:53:41.344 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: mgrmap e7: y(active, since 1.00568s) 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-mon[29803]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: 2024-09-16T08:53:41.174+0000 7f97ea82f640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:08:53:41] ENGINE Error in HTTPServer.serve 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: Traceback (most recent call last): 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self._connections.run(self.expiration_interval) 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T08:53:41.345 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self._run(expiration_interval) 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: s = self.context.wrap_socket( 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: return self.sslsocket_class._create( 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self.do_handshake() 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self._sslobj.do_handshake() 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T08:53:41.346 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:41 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: 2024-09-16T08:53:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-09-16T08:53:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-16T08:53:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T08:53:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-09-16T08:53:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.482+0000 7f002572c640 1 Processor -- start 2024-09-16T08:53:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.482+0000 7f002572c640 1 -- start start 2024-09-16T08:53:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.482+0000 7f002572c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f00201091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.482+0000 7f002572c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0020109730 con 0x7f002007a410 2024-09-16T08:53:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.482+0000 7f001effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f00201091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.483+0000 7f001effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f00201091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44002/0 (socket says 172.21.15.111:44002) 2024-09-16T08:53:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.483+0000 7f001effd640 1 -- 172.21.15.111:0/2311183040 learned_addr learned my addr 172.21.15.111:0/2311183040 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.483+0000 7f001effd640 1 -- 172.21.15.111:0/2311183040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00201098b0 con 0x7f002007a410 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.484+0000 7f001effd640 1 --2- 172.21.15.111:0/2311183040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f00201091f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0010009820 tx=0x7f001002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310c41a2f32217ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.484+0000 7f001e7fc640 1 -- 172.21.15.111:0/2311183040 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0010037070 con 0x7f002007a410 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.484+0000 7f001e7fc640 1 -- 172.21.15.111:0/2311183040 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f0010032870 con 0x7f002007a410 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.484+0000 7f001e7fc640 1 -- 172.21.15.111:0/2311183040 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00100305d0 con 0x7f002007a410 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.485+0000 7f002572c640 1 -- 172.21.15.111:0/2311183040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 msgr2=0x7f00201091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.485+0000 7f002572c640 1 --2- 172.21.15.111:0/2311183040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f00201091f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0010009820 tx=0x7f001002a270 comp rx=0 tx=0).stop 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.485+0000 7f002572c640 1 -- 172.21.15.111:0/2311183040 shutdown_connections 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.485+0000 7f002572c640 1 --2- 172.21.15.111:0/2311183040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f00201091f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.485+0000 7f002572c640 1 -- 172.21.15.111:0/2311183040 >> 172.21.15.111:0/2311183040 conn(0x7f0020100e00 msgr2=0x7f0020103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.485+0000 7f002572c640 1 -- 172.21.15.111:0/2311183040 shutdown_connections 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.485+0000 7f002572c640 1 -- 172.21.15.111:0/2311183040 wait complete. 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.485+0000 7f002572c640 1 Processor -- start 2024-09-16T08:53:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.486+0000 7f002572c640 1 -- start start 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.486+0000 7f002572c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f002019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.486+0000 7f002572c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f002010a8d0 con 0x7f002007a410 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.486+0000 7f001effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f002019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.486+0000 7f001effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f002019a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44012/0 (socket says 172.21.15.111:44012) 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.486+0000 7f001effd640 1 -- 172.21.15.111:0/2174159387 learned_addr learned my addr 172.21.15.111:0/2174159387 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.486+0000 7f001effd640 1 -- 172.21.15.111:0/2174159387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002019ab90 con 0x7f002007a410 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.486+0000 7f001effd640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f002019a630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f00100097f0 tx=0x7f0010030b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.487+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f001003f020 con 0x7f002007a410 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.487+0000 7f002572c640 1 -- 172.21.15.111:0/2174159387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f002019ae20 con 0x7f002007a410 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.487+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f0010030e40 con 0x7f002007a410 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.487+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0010037040 con 0x7f002007a410 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.487+0000 7f002572c640 1 -- 172.21.15.111:0/2174159387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f002019b7e0 con 0x7f002007a410 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.488+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) ==== 51765+0+0 (secure 0 0 0) 0x7f0010042020 con 0x7f002007a410 2024-09-16T08:53:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.488+0000 7f001cff9640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 0x7f000c045f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.488+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 --> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f000c046630 con 0x7f000c043a60 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.488+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0010074250 con 0x7f002007a410 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.488+0000 7f0017fff640 1 -- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 msgr2=0x7f000c045f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2026406736 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.488+0000 7f0017fff640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 0x7f000c045f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.690+0000 7f0017fff640 1 -- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 msgr2=0x7f000c045f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2026406736 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:37.690+0000 7f0017fff640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 0x7f000c045f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:38.091+0000 7f0017fff640 1 -- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 msgr2=0x7f000c045f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2026406736 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:38.091+0000 7f0017fff640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 0x7f000c045f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:38.892+0000 7f0017fff640 1 -- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 msgr2=0x7f000c045f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2026406736 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:38.892+0000 7f0017fff640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 0x7f000c045f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:39.767+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mgrmap(e 6) ==== 51532+0+0 (secure 0 0 0) 0x7f0010073050 con 0x7f002007a410 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:39.767+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 msgr2=0x7f000c045f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:53:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:39.767+0000 7f001cff9640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 0x7f000c045f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.768+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 7) ==== 51659+0+0 (secure 0 0 0) 0x7f001003b410 con 0x7f002007a410 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.768+0000 7f001cff9640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f000c047350 0x7f000c049740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.769+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0010040ec0 con 0x7f000c047350 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.769+0000 7f0017fff640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f000c047350 0x7f000c049740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.769+0000 7f0017fff640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f000c047350 0x7f000c049740 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0008003980 tx=0x7f0008006e10 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.770+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 <== mgr.14118 v2:172.21.15.111:6800/2366462893 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f0010040ec0 con 0x7f000c047350 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.777+0000 7f002572c640 1 -- 172.21.15.111:0/2174159387 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f0020107260 con 0x7f000c047350 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.777+0000 7f001cff9640 1 -- 172.21.15.111:0/2174159387 <== mgr.14118 v2:172.21.15.111:6800/2366462893 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f0020107260 con 0x7f000c047350 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.777+0000 7f002572c640 1 -- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f000c047350 msgr2=0x7f000c049740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.777+0000 7f002572c640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f000c047350 0x7f000c049740 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0008003980 tx=0x7f0008006e10 comp rx=0 tx=0).stop 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.777+0000 7f002572c640 1 -- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 msgr2=0x7f002019a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.777+0000 7f002572c640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f002019a630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f00100097f0 tx=0x7f0010030b00 comp rx=0 tx=0).stop 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.778+0000 7f002572c640 1 -- 172.21.15.111:0/2174159387 shutdown_connections 2024-09-16T08:53:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.778+0000 7f002572c640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f000c047350 0x7f000c049740 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.778+0000 7f002572c640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:6800/2026406736,v1:172.21.15.111:6801/2026406736] conn(0x7f000c043a60 0x7f000c045f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.778+0000 7f002572c640 1 --2- 172.21.15.111:0/2174159387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f002007a410 0x7f002019a630 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.778+0000 7f002572c640 1 -- 172.21.15.111:0/2174159387 >> 172.21.15.111:0/2174159387 conn(0x7f0020100e00 msgr2=0x7f00201019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.778+0000 7f002572c640 1 -- 172.21.15.111:0/2174159387 shutdown_connections 2024-09-16T08:53:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:40.778+0000 7f002572c640 1 -- 172.21.15.111:0/2174159387 wait complete. 2024-09-16T08:53:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:mgr epoch 5 is available 2024-09-16T08:53:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying orchestrator module is enabled... 2024-09-16T08:53:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:42 smithi111 ceph-mon[29803]: [16/Sep/2024:08:53:41] ENGINE Bus STARTING 2024-09-16T08:53:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:42 smithi111 ceph-mon[29803]: [16/Sep/2024:08:53:41] ENGINE Serving on https://172.21.15.111:7150 2024-09-16T08:53:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:42 smithi111 ceph-mon[29803]: [16/Sep/2024:08:53:41] ENGINE Error in HTTPServer.serve 2024-09-16T08:53:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout: Traceback (most recent call last): 2024-09-16T08:53:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T08:53:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout: self._connections.run(self.expiration_interval) 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: self._run(expiration_interval) 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: s = self.context.wrap_socket( 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: return self.sslsocket_class._create( 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: self.do_handshake() 2024-09-16T08:53:42.427 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T08:53:42.428 INFO:journalctl@ceph.mon.a.smithi111.stdout: self._sslobj.do_handshake() 2024-09-16T08:53:42.428 INFO:journalctl@ceph.mon.a.smithi111.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T08:53:42.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:42 smithi111 ceph-mon[29803]: [16/Sep/2024:08:53:41] ENGINE Serving on http://172.21.15.111:8765 2024-09-16T08:53:42.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:42 smithi111 ceph-mon[29803]: [16/Sep/2024:08:53:41] ENGINE Bus STARTED 2024-09-16T08:53:42.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:42 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:53:42.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/418432970' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-16T08:53:42.808 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.784+0000 7fa6aedba640 1 Processor -- start 2024-09-16T08:53:42.808 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.784+0000 7fa6aedba640 1 -- start start 2024-09-16T08:53:42.808 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.785+0000 7fa6aedba640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a8106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:42.808 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.785+0000 7fa6aedba640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8107000 con 0x7fa6a8106610 2024-09-16T08:53:42.808 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.785+0000 7fa6acb56640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a8106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:42.808 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.785+0000 7fa6acb56640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a8106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46910/0 (socket says 172.21.15.111:46910) 2024-09-16T08:53:42.808 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.785+0000 7fa6acb56640 1 -- 172.21.15.111:0/731876622 learned_addr learned my addr 172.21.15.111:0/731876622 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.785+0000 7fa6acb56640 1 -- 172.21.15.111:0/731876622 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a8107180 con 0x7fa6a8106610 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.786+0000 7fa6acb56640 1 --2- 172.21.15.111:0/731876622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a8106a10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009820 tx=0x7fa69c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c44d10c830c294 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.786+0000 7fa6a77fe640 1 -- 172.21.15.111:0/731876622 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa69c037070 con 0x7fa6a8106610 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.786+0000 7fa6a77fe640 1 -- 172.21.15.111:0/731876622 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fa69c032870 con 0x7fa6a8106610 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.786+0000 7fa6a77fe640 1 -- 172.21.15.111:0/731876622 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa69c0305d0 con 0x7fa6a8106610 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.786+0000 7fa6aedba640 1 -- 172.21.15.111:0/731876622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 msgr2=0x7fa6a8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.786+0000 7fa6aedba640 1 --2- 172.21.15.111:0/731876622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a8106a10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009820 tx=0x7fa69c02a270 comp rx=0 tx=0).stop 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.786+0000 7fa6aedba640 1 -- 172.21.15.111:0/731876622 shutdown_connections 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.786+0000 7fa6aedba640 1 --2- 172.21.15.111:0/731876622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a8106a10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.786+0000 7fa6aedba640 1 -- 172.21.15.111:0/731876622 >> 172.21.15.111:0/731876622 conn(0x7fa6a8101f00 msgr2=0x7fa6a8104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.787+0000 7fa6aedba640 1 -- 172.21.15.111:0/731876622 shutdown_connections 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.787+0000 7fa6aedba640 1 -- 172.21.15.111:0/731876622 wait complete. 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.787+0000 7fa6aedba640 1 Processor -- start 2024-09-16T08:53:42.809 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.787+0000 7fa6aedba640 1 -- start start 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.787+0000 7fa6aedba640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.787+0000 7fa6aedba640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a810aa90 con 0x7fa6a8106610 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.787+0000 7fa6acb56640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.787+0000 7fa6acb56640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46916/0 (socket says 172.21.15.111:46916) 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.787+0000 7fa6acb56640 1 -- 172.21.15.111:0/418432970 learned_addr learned my addr 172.21.15.111:0/418432970 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.788+0000 7fa6acb56640 1 -- 172.21.15.111:0/418432970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a819abb0 con 0x7fa6a8106610 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.788+0000 7fa6acb56640 1 --2- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a819a650 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa69c0090b0 tx=0x7fa69c0307c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.788+0000 7fa6a5ffb640 1 -- 172.21.15.111:0/418432970 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa69c03f020 con 0x7fa6a8106610 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.788+0000 7fa6a5ffb640 1 -- 172.21.15.111:0/418432970 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fa69c030e20 con 0x7fa6a8106610 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.788+0000 7fa6a5ffb640 1 -- 172.21.15.111:0/418432970 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa69c037040 con 0x7fa6a8106610 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.788+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a819ae40 con 0x7fa6a8106610 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.788+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a819b800 con 0x7fa6a8106610 2024-09-16T08:53:42.810 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.789+0000 7fa6a5ffb640 1 -- 172.21.15.111:0/418432970 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 7) ==== 51659+0+0 (secure 0 0 0) 0x7fa69c042020 con 0x7fa6a8106610 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.789+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a810ac10 con 0x7fa6a8106610 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.789+0000 7fa6a5ffb640 1 --2- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fa69803f590 0x7fa698041a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.789+0000 7fa6a5ffb640 1 -- 172.21.15.111:0/418432970 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa69c0743f0 con 0x7fa6a8106610 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.789+0000 7fa6a7fff640 1 --2- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fa69803f590 0x7fa698041a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.790+0000 7fa6a7fff640 1 --2- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fa69803f590 0x7fa698041a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa6940098c0 tx=0x7fa694006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:41.794+0000 7fa6a5ffb640 1 -- 172.21.15.111:0/418432970 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fa69c049020 con 0x7fa6a8106610 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.013+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "orchestrator"} v 0) -- 0x7fa6a819beb0 con 0x7fa6a8106610 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.239+0000 7fa6a5ffb640 1 -- 172.21.15.111:0/418432970 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "orchestrator"}]=0 module 'orchestrator' is already enabled (always-on) v8) ==== 143+0+0 (secure 0 0 0) 0x7fa69c03f1c0 con 0x7fa6a8106610 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.240+0000 7fa6a5ffb640 1 -- 172.21.15.111:0/418432970 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7fa69c049cf0 con 0x7fa6a8106610 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fa69803f590 msgr2=0x7fa698041a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 --2- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fa69803f590 0x7fa698041a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa6940098c0 tx=0x7fa694006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 msgr2=0x7fa6a819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:42.811 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 --2- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a819a650 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa69c0090b0 tx=0x7fa69c0307c0 comp rx=0 tx=0).stop 2024-09-16T08:53:42.812 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 shutdown_connections 2024-09-16T08:53:42.812 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 --2- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fa69803f590 0x7fa698041a50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:42.812 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 --2- 172.21.15.111:0/418432970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6a8106610 0x7fa6a819a650 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:42.812 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 >> 172.21.15.111:0/418432970 conn(0x7fa6a8101f00 msgr2=0x7fa6a8102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:42.812 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 shutdown_connections 2024-09-16T08:53:42.812 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:42.242+0000 7fa6aedba640 1 -- 172.21.15.111:0/418432970 wait complete. 2024-09-16T08:53:42.812 INFO:teuthology.orchestra.run.smithi111.stdout:Setting orchestrator backend to cephadm... 2024-09-16T08:53:43.425 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/418432970' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-16T08:53:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:43 smithi111 ceph-mon[29803]: mgrmap e8: y(active, since 2s) 2024-09-16T08:53:44.181 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.248+0000 7fdc10b5f640 1 Processor -- start 2024-09-16T08:53:44.182 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.248+0000 7fdc10b5f640 1 -- start start 2024-09-16T08:53:44.182 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.248+0000 7fdc10b5f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:44.182 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.248+0000 7fdc10b5f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0c107000 con 0x7fdc0c106610 2024-09-16T08:53:44.182 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.249+0000 7fdc0a59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:44.182 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.249+0000 7fdc0a59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46918/0 (socket says 172.21.15.111:46918) 2024-09-16T08:53:44.182 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.249+0000 7fdc0a59c640 1 -- 172.21.15.111:0/1032117824 learned_addr learned my addr 172.21.15.111:0/1032117824 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:44.182 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.249+0000 7fdc0a59c640 1 -- 172.21.15.111:0/1032117824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0c107180 con 0x7fdc0c106610 2024-09-16T08:53:44.183 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.249+0000 7fdc0a59c640 1 --2- 172.21.15.111:0/1032117824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c106a10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009e80 tx=0x7fdbfc02a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2fcd865ffd67a74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:44.183 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc0959a640 1 -- 172.21.15.111:0/1032117824 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc036070 con 0x7fdc0c106610 2024-09-16T08:53:44.183 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc0959a640 1 -- 172.21.15.111:0/1032117824 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fdbfc031830 con 0x7fdc0c106610 2024-09-16T08:53:44.183 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc0959a640 1 -- 172.21.15.111:0/1032117824 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc007780 con 0x7fdc0c106610 2024-09-16T08:53:44.183 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc10b5f640 1 -- 172.21.15.111:0/1032117824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 msgr2=0x7fdc0c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:44.183 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc10b5f640 1 --2- 172.21.15.111:0/1032117824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c106a10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009e80 tx=0x7fdbfc02a7a0 comp rx=0 tx=0).stop 2024-09-16T08:53:44.183 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc10b5f640 1 -- 172.21.15.111:0/1032117824 shutdown_connections 2024-09-16T08:53:44.183 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc10b5f640 1 --2- 172.21.15.111:0/1032117824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c106a10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:44.184 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc10b5f640 1 -- 172.21.15.111:0/1032117824 >> 172.21.15.111:0/1032117824 conn(0x7fdc0c101f00 msgr2=0x7fdc0c104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:44.184 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc10b5f640 1 -- 172.21.15.111:0/1032117824 shutdown_connections 2024-09-16T08:53:44.184 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.250+0000 7fdc10b5f640 1 -- 172.21.15.111:0/1032117824 wait complete. 2024-09-16T08:53:44.184 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.251+0000 7fdc10b5f640 1 Processor -- start 2024-09-16T08:53:44.184 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.251+0000 7fdc10b5f640 1 -- start start 2024-09-16T08:53:44.184 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.251+0000 7fdc10b5f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:44.184 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.251+0000 7fdc10b5f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0c10aa90 con 0x7fdc0c106610 2024-09-16T08:53:44.185 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.251+0000 7fdc0a59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:44.185 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.251+0000 7fdc0a59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c19a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46932/0 (socket says 172.21.15.111:46932) 2024-09-16T08:53:44.185 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.251+0000 7fdc0a59c640 1 -- 172.21.15.111:0/3082811466 learned_addr learned my addr 172.21.15.111:0/3082811466 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:44.185 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.252+0000 7fdc0a59c640 1 -- 172.21.15.111:0/3082811466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0c19abb0 con 0x7fdc0c106610 2024-09-16T08:53:44.185 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.252+0000 7fdc0a59c640 1 --2- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c19a650 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009fa0 tx=0x7fdbfc0077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:44.185 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.252+0000 7fdc037fe640 1 -- 172.21.15.111:0/3082811466 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc041020 con 0x7fdc0c106610 2024-09-16T08:53:44.185 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.252+0000 7fdc037fe640 1 -- 172.21.15.111:0/3082811466 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fdbfc03c470 con 0x7fdc0c106610 2024-09-16T08:53:44.185 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.252+0000 7fdc037fe640 1 -- 172.21.15.111:0/3082811466 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc036040 con 0x7fdc0c106610 2024-09-16T08:53:44.185 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.252+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0c19ae40 con 0x7fdc0c106610 2024-09-16T08:53:44.186 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.252+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc0c19b880 con 0x7fdc0c106610 2024-09-16T08:53:44.186 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.253+0000 7fdc037fe640 1 -- 172.21.15.111:0/3082811466 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7fdbfc044020 con 0x7fdc0c106610 2024-09-16T08:53:44.186 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.253+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc0c10ac10 con 0x7fdc0c106610 2024-09-16T08:53:44.186 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.253+0000 7fdc037fe640 1 --2- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fdbf803f6a0 0x7fdbf8041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:44.186 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.253+0000 7fdc037fe640 1 -- 172.21.15.111:0/3082811466 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdbfc075010 con 0x7fdc0c106610 2024-09-16T08:53:44.186 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.253+0000 7fdc09d9b640 1 --2- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fdbf803f6a0 0x7fdbf8041b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:44.186 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.254+0000 7fdc09d9b640 1 --2- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fdbf803f6a0 0x7fdbf8041b60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40098c0 tx=0x7fdbf4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:44.186 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.258+0000 7fdc037fe640 1 -- 172.21.15.111:0/3082811466 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fdbfc04b020 con 0x7fdc0c106610 2024-09-16T08:53:44.187 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.448+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fdc0c1031e0 con 0x7fdbf803f6a0 2024-09-16T08:53:44.187 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.453+0000 7fdc037fe640 1 -- 172.21.15.111:0/3082811466 <== mgr.14118 v2:172.21.15.111:6800/2366462893 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdc0c1031e0 con 0x7fdbf803f6a0 2024-09-16T08:53:44.187 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fdbf803f6a0 msgr2=0x7fdbf8041b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:44.187 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 --2- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fdbf803f6a0 0x7fdbf8041b60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40098c0 tx=0x7fdbf4006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:44.187 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 msgr2=0x7fdc0c19a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:44.187 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 --2- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c19a650 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009fa0 tx=0x7fdbfc0077a0 comp rx=0 tx=0).stop 2024-09-16T08:53:44.187 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 shutdown_connections 2024-09-16T08:53:44.188 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 --2- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fdbf803f6a0 0x7fdbf8041b60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:44.188 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 --2- 172.21.15.111:0/3082811466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc0c106610 0x7fdc0c19a650 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:44.188 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 >> 172.21.15.111:0/3082811466 conn(0x7fdc0c101f00 msgr2=0x7fdc0c102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:44.188 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 shutdown_connections 2024-09-16T08:53:44.188 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:43.455+0000 7fdc10b5f640 1 -- 172.21.15.111:0/3082811466 wait complete. 2024-09-16T08:53:44.818 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:44 smithi111 ceph-mon[29803]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:53:44.818 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:44 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:44.818 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:44 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:53:45.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.634+0000 7f9a859af640 1 Processor -- start 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.634+0000 7f9a859af640 1 -- start start 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.634+0000 7f9a859af640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a801058f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.634+0000 7f9a859af640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9a80105ee0 con 0x7f9a801054f0 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.634+0000 7f9a7effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a801058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.634+0000 7f9a7effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a801058f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46944/0 (socket says 172.21.15.111:46944) 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.634+0000 7f9a7effd640 1 -- 172.21.15.111:0/338299021 learned_addr learned my addr 172.21.15.111:0/338299021 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.634+0000 7f9a7effd640 1 -- 172.21.15.111:0/338299021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a80106060 con 0x7f9a801054f0 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.635+0000 7f9a7effd640 1 --2- 172.21.15.111:0/338299021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a801058f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009820 tx=0x7f9a7002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5421cf959f325cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.635+0000 7f9a7dffb640 1 -- 172.21.15.111:0/338299021 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a70037070 con 0x7f9a801054f0 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.635+0000 7f9a7dffb640 1 -- 172.21.15.111:0/338299021 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f9a70032870 con 0x7f9a801054f0 2024-09-16T08:53:45.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.635+0000 7f9a7dffb640 1 -- 172.21.15.111:0/338299021 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a700305d0 con 0x7f9a801054f0 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.636+0000 7f9a859af640 1 -- 172.21.15.111:0/338299021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 msgr2=0x7f9a801058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.636+0000 7f9a859af640 1 --2- 172.21.15.111:0/338299021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a801058f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009820 tx=0x7f9a7002a270 comp rx=0 tx=0).stop 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.636+0000 7f9a859af640 1 -- 172.21.15.111:0/338299021 shutdown_connections 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.636+0000 7f9a859af640 1 --2- 172.21.15.111:0/338299021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a801058f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.636+0000 7f9a859af640 1 -- 172.21.15.111:0/338299021 >> 172.21.15.111:0/338299021 conn(0x7f9a80100e00 msgr2=0x7f9a80103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.636+0000 7f9a859af640 1 -- 172.21.15.111:0/338299021 shutdown_connections 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.636+0000 7f9a859af640 1 -- 172.21.15.111:0/338299021 wait complete. 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.636+0000 7f9a859af640 1 Processor -- start 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.636+0000 7f9a859af640 1 -- start start 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a859af640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a8019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:45.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a859af640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9a80109970 con 0x7f9a801054f0 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a7effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a8019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a7effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a8019a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46954/0 (socket says 172.21.15.111:46954) 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a7effd640 1 -- 172.21.15.111:0/1061186243 learned_addr learned my addr 172.21.15.111:0/1061186243 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a7effd640 1 -- 172.21.15.111:0/1061186243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8019ab50 con 0x7f9a801054f0 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a7effd640 1 --2- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a8019a5f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9a700097f0 tx=0x7f9a70030b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a849ad640 1 -- 172.21.15.111:0/1061186243 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a7003f020 con 0x7f9a801054f0 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a849ad640 1 -- 172.21.15.111:0/1061186243 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f9a70030e40 con 0x7f9a801054f0 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a849ad640 1 -- 172.21.15.111:0/1061186243 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a70037040 con 0x7f9a801054f0 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8019ade0 con 0x7f9a801054f0 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.637+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8019b7a0 con 0x7f9a801054f0 2024-09-16T08:53:45.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.638+0000 7f9a849ad640 1 -- 172.21.15.111:0/1061186243 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7f9a70042020 con 0x7f9a801054f0 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.638+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a80109af0 con 0x7f9a801054f0 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.639+0000 7f9a849ad640 1 --2- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f9a6c03f6a0 0x7f9a6c041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.639+0000 7f9a849ad640 1 -- 172.21.15.111:0/1061186243 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9a70074590 con 0x7f9a801054f0 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.639+0000 7f9a7e7fc640 1 --2- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f9a6c03f6a0 0x7f9a6c041b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.639+0000 7f9a7e7fc640 1 --2- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f9a6c03f6a0 0x7f9a6c041b60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9a680098c0 tx=0x7f9a68006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.643+0000 7f9a849ad640 1 -- 172.21.15.111:0/1061186243 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f9a70049020 con 0x7f9a801054f0 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.817+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f9a80079f50 con 0x7f9a6c03f6a0 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.817+0000 7f9a849ad640 1 -- 172.21.15.111:0/1061186243 <== mgr.14118 v2:172.21.15.111:6800/2366462893 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f9a80079f50 con 0x7f9a6c03f6a0 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f9a6c03f6a0 msgr2=0x7f9a6c041b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 --2- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f9a6c03f6a0 0x7f9a6c041b60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9a680098c0 tx=0x7f9a68006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 msgr2=0x7f9a8019a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 --2- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a8019a5f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9a700097f0 tx=0x7f9a70030b00 comp rx=0 tx=0).stop 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 shutdown_connections 2024-09-16T08:53:45.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 --2- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f9a6c03f6a0 0x7f9a6c041b60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:45.451 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 --2- 172.21.15.111:0/1061186243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a801054f0 0x7f9a8019a5f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:45.451 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 >> 172.21.15.111:0/1061186243 conn(0x7f9a80100e00 msgr2=0x7f9a80079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:45.451 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 shutdown_connections 2024-09-16T08:53:45.451 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:44.820+0000 7f9a859af640 1 -- 172.21.15.111:0/1061186243 wait complete. 2024-09-16T08:53:45.451 INFO:teuthology.orchestra.run.smithi111.stdout:Generating ssh key... 2024-09-16T08:53:46.031 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:45 smithi111 ceph-mon[29803]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:53:46.652 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: Generating public/private rsa key pair. 2024-09-16T08:53:46.652 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: Your identification has been saved in /tmp/tmpkryha9x_/key 2024-09-16T08:53:46.652 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: Your public key has been saved in /tmp/tmpkryha9x_/key.pub 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: The key fingerprint is: 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: SHA256:H6ZhcahIxo2iHh4MkBsJtWxW1C38FZWMRiIQvA4jPUI ceph-f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: The key's randomart image is: 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: +---[RSA 3072]----+ 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: |++.o==....o=.. | 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: |=E +.o+..o+ o | 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: |o+* =..ooo. | 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: |*+=+.. ..o | 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: |.* =. . S o | 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: |o o . . = . | 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: | o . . | 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: | | 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: | | 2024-09-16T08:53:46.653 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:53:46 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: +----[SHA256]-----+ 2024-09-16T08:53:46.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.842+0000 7ff77c6b2640 1 Processor -- start 2024-09-16T08:53:46.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.842+0000 7ff77c6b2640 1 -- start start 2024-09-16T08:53:46.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.842+0000 7ff77c6b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff774106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.842+0000 7ff77c6b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff774107000 con 0x7ff774106610 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.842+0000 7ff77a44e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff774106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.842+0000 7ff77a44e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff774106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46964/0 (socket says 172.21.15.111:46964) 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.842+0000 7ff77a44e640 1 -- 172.21.15.111:0/4092391205 learned_addr learned my addr 172.21.15.111:0/4092391205 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.843+0000 7ff77a44e640 1 -- 172.21.15.111:0/4092391205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff774107180 con 0x7ff774106610 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.843+0000 7ff77a44e640 1 --2- 172.21.15.111:0/4092391205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff774106a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff770009820 tx=0x7ff77002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e2afc4d378f60c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.843+0000 7ff77944c640 1 -- 172.21.15.111:0/4092391205 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff770037070 con 0x7ff774106610 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.843+0000 7ff77944c640 1 -- 172.21.15.111:0/4092391205 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7ff770032870 con 0x7ff774106610 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.843+0000 7ff77944c640 1 -- 172.21.15.111:0/4092391205 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7700305d0 con 0x7ff774106610 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.844+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4092391205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 msgr2=0x7ff774106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.844+0000 7ff77c6b2640 1 --2- 172.21.15.111:0/4092391205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff774106a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff770009820 tx=0x7ff77002a270 comp rx=0 tx=0).stop 2024-09-16T08:53:46.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.844+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4092391205 shutdown_connections 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.844+0000 7ff77c6b2640 1 --2- 172.21.15.111:0/4092391205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff774106a10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.844+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4092391205 >> 172.21.15.111:0/4092391205 conn(0x7ff774101f00 msgr2=0x7ff774104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.844+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4092391205 shutdown_connections 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.844+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4092391205 wait complete. 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.844+0000 7ff77c6b2640 1 Processor -- start 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77c6b2640 1 -- start start 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77c6b2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff77419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77c6b2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff77410aa90 con 0x7ff774106610 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77a44e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff77419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77a44e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff77419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46978/0 (socket says 172.21.15.111:46978) 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77a44e640 1 -- 172.21.15.111:0/4226972847 learned_addr learned my addr 172.21.15.111:0/4226972847 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77a44e640 1 -- 172.21.15.111:0/4226972847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77419abb0 con 0x7ff774106610 2024-09-16T08:53:46.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77a44e640 1 --2- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff77419a650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff7700097f0 tx=0x7ff770030b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff76b7fe640 1 -- 172.21.15.111:0/4226972847 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff77003f020 con 0x7ff774106610 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff76b7fe640 1 -- 172.21.15.111:0/4226972847 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7ff770030e40 con 0x7ff774106610 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff77419ae40 con 0x7ff774106610 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff76b7fe640 1 -- 172.21.15.111:0/4226972847 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff770037040 con 0x7ff774106610 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.845+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff77419b800 con 0x7ff774106610 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.846+0000 7ff76b7fe640 1 -- 172.21.15.111:0/4226972847 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7ff770042020 con 0x7ff774106610 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.846+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff77410ac10 con 0x7ff774106610 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.846+0000 7ff76b7fe640 1 --2- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff76003f6a0 0x7ff760041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.846+0000 7ff76b7fe640 1 -- 172.21.15.111:0/4226972847 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff770074590 con 0x7ff774106610 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.847+0000 7ff779c4d640 1 --2- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff76003f6a0 0x7ff760041b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.847+0000 7ff779c4d640 1 --2- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff76003f6a0 0x7ff760041b60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff7640098c0 tx=0x7ff764006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:45.851+0000 7ff76b7fe640 1 -- 172.21.15.111:0/4226972847 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7ff770049020 con 0x7ff774106610 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.029+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7ff7741031e0 con 0x7ff76003f6a0 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.294+0000 7ff76b7fe640 1 -- 172.21.15.111:0/4226972847 <== mgr.14118 v2:172.21.15.111:6800/2366462893 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff7741031e0 con 0x7ff76003f6a0 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff76003f6a0 msgr2=0x7ff760041b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 --2- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff76003f6a0 0x7ff760041b60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff7640098c0 tx=0x7ff764006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 msgr2=0x7ff77419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 --2- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff77419a650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff7700097f0 tx=0x7ff770030b00 comp rx=0 tx=0).stop 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 shutdown_connections 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 --2- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff76003f6a0 0x7ff760041b60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 --2- 172.21.15.111:0/4226972847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff774106610 0x7ff77419a650 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:46.949 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 >> 172.21.15.111:0/4226972847 conn(0x7ff774101f00 msgr2=0x7ff774102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:46.949 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 shutdown_connections 2024-09-16T08:53:46.949 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:46.297+0000 7ff77c6b2640 1 -- 172.21.15.111:0/4226972847 wait complete. 2024-09-16T08:53:47.562 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:47 smithi111 ceph-mon[29803]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:53:47.562 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:47 smithi111 ceph-mon[29803]: Generating ssh key... 2024-09-16T08:53:47.562 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:47 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:47.562 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:47 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:48.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5VnMGKtwHDkwCxDpc3F9Xr0xhgkhmDDlYaz+iGkUhJLHNkOxsU7sgKobQZ3Rx/CkjMe6mE6FkGtTx1scjSC25nEkRvjXUKd4eIBW5XucVEPOYezXnglh/U65v1gdLHJUZJcqW3KiQehqIHrqfQFtSKT41i8L/5Y+UL9DklOPaKpQUiyvRZW5O7RGufGOO6xcUpAUNKYwCLlkiiEV4Ru8Fs21gpFLYuthcpN/n0B18brnzNvF92lsxS6w+rggIEabeDS1Pw9vD0Rovu9qtjC6qbANf0Ju0yGNN68W94kGsZgq0UM+ydJsk+/jZhNMQipoXKL453pDX+H2IUTqc7zdGfX8UJz/VnFZHQt6HbhuFP0kxevVt96zpAa/GK3TvPcPbHAgDLSpORhBP8wr5Rj5YsUPzvPQ3H/KcG5BQKZDSZIQDs15Ph9qHDWEagXEEiRf3vzGxZb5V4dIJex6yFTYvbi5eG1ggAqZCzxsUbnPY9pbNP+HWb7HjErD5nHKjyLE= ceph-f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:53:48.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.372+0000 7fe8372d4640 1 Processor -- start 2024-09-16T08:53:48.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.373+0000 7fe8372d4640 1 -- start start 2024-09-16T08:53:48.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.373+0000 7fe8372d4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83010b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:48.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.373+0000 7fe8372d4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe83007a6e0 con 0x7fe83007a2e0 2024-09-16T08:53:48.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.373+0000 7fe835070640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83010b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:48.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.373+0000 7fe835070640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83010b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46982/0 (socket says 172.21.15.111:46982) 2024-09-16T08:53:48.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.373+0000 7fe835070640 1 -- 172.21.15.111:0/74725277 learned_addr learned my addr 172.21.15.111:0/74725277 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:48.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.373+0000 7fe835070640 1 -- 172.21.15.111:0/74725277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83010bb70 con 0x7fe83007a2e0 2024-09-16T08:53:48.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.374+0000 7fe835070640 1 --2- 172.21.15.111:0/74725277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83010b630 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009820 tx=0x7fe82c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2477b16bcc24fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:48.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.374+0000 7fe827fff640 1 -- 172.21.15.111:0/74725277 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe82c037070 con 0x7fe83007a2e0 2024-09-16T08:53:48.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.374+0000 7fe827fff640 1 -- 172.21.15.111:0/74725277 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fe82c032870 con 0x7fe83007a2e0 2024-09-16T08:53:48.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.374+0000 7fe827fff640 1 -- 172.21.15.111:0/74725277 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe82c0305d0 con 0x7fe83007a2e0 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 -- 172.21.15.111:0/74725277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 msgr2=0x7fe83010b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 --2- 172.21.15.111:0/74725277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83010b630 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009820 tx=0x7fe82c02a270 comp rx=0 tx=0).stop 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 -- 172.21.15.111:0/74725277 shutdown_connections 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 --2- 172.21.15.111:0/74725277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83010b630 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 -- 172.21.15.111:0/74725277 >> 172.21.15.111:0/74725277 conn(0x7fe830100e00 msgr2=0x7fe830103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 -- 172.21.15.111:0/74725277 shutdown_connections 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 -- 172.21.15.111:0/74725277 wait complete. 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 Processor -- start 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 -- start start 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.375+0000 7fe8372d4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe83010cb30 con 0x7fe83007a2e0 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe835070640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:48.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe835070640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46984/0 (socket says 172.21.15.111:46984) 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe835070640 1 -- 172.21.15.111:0/961310075 learned_addr learned my addr 172.21.15.111:0/961310075 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe835070640 1 -- 172.21.15.111:0/961310075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83019cdc0 con 0x7fe83007a2e0 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe835070640 1 --2- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83019c860 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0090b0 tx=0x7fe82c0307c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe8267fc640 1 -- 172.21.15.111:0/961310075 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe82c03f020 con 0x7fe83007a2e0 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe8267fc640 1 -- 172.21.15.111:0/961310075 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7fe82c030e20 con 0x7fe83007a2e0 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83019d050 con 0x7fe83007a2e0 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe8267fc640 1 -- 172.21.15.111:0/961310075 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe82c037040 con 0x7fe83007a2e0 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.376+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83019da10 con 0x7fe83007a2e0 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.377+0000 7fe8267fc640 1 -- 172.21.15.111:0/961310075 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7fe82c042020 con 0x7fe83007a2e0 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.377+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe83010b6b0 con 0x7fe83007a2e0 2024-09-16T08:53:48.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.377+0000 7fe8267fc640 1 --2- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fe81c03f6a0 0x7fe81c041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.377+0000 7fe8267fc640 1 -- 172.21.15.111:0/961310075 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe82c074610 con 0x7fe83007a2e0 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.377+0000 7fe83486f640 1 --2- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fe81c03f6a0 0x7fe81c041b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.378+0000 7fe83486f640 1 --2- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fe81c03f6a0 0x7fe81c041b60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe8200098c0 tx=0x7fe820006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.382+0000 7fe8267fc640 1 -- 172.21.15.111:0/961310075 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fe82c049020 con 0x7fe83007a2e0 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.560+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fe8301020e0 con 0x7fe81c03f6a0 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.561+0000 7fe8267fc640 1 -- 172.21.15.111:0/961310075 <== mgr.14118 v2:172.21.15.111:6800/2366462893 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fe8301020e0 con 0x7fe81c03f6a0 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fe81c03f6a0 msgr2=0x7fe81c041b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 --2- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fe81c03f6a0 0x7fe81c041b60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe8200098c0 tx=0x7fe820006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 msgr2=0x7fe83019c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 --2- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83019c860 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0090b0 tx=0x7fe82c0307c0 comp rx=0 tx=0).stop 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 shutdown_connections 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 --2- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fe81c03f6a0 0x7fe81c041b60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:48.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 --2- 172.21.15.111:0/961310075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe83007a2e0 0x7fe83019c860 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:48.157 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 >> 172.21.15.111:0/961310075 conn(0x7fe830100e00 msgr2=0x7fe8301019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:48.157 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 shutdown_connections 2024-09-16T08:53:48.157 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:47.563+0000 7fe8372d4640 1 -- 172.21.15.111:0/961310075 wait complete. 2024-09-16T08:53:48.157 INFO:teuthology.orchestra.run.smithi111.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-16T08:53:48.157 INFO:teuthology.orchestra.run.smithi111.stdout:Adding key to root@localhost authorized_keys... 2024-09-16T08:53:48.157 INFO:teuthology.orchestra.run.smithi111.stdout:Adding host smithi111... 2024-09-16T08:53:48.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:48 smithi111 ceph-mon[29803]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:53:49.513 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:49 smithi111 ceph-mon[29803]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "addr": "172.21.15.111", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:53:50.425 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:50 smithi111 ceph-mon[29803]: Deploying cephadm binary to smithi111 2024-09-16T08:53:52.407 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Added host 'smithi111' with addr '172.21.15.111' 2024-09-16T08:53:52.407 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.557+0000 7f1bda050640 1 Processor -- start 2024-09-16T08:53:52.407 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.557+0000 7f1bda050640 1 -- start start 2024-09-16T08:53:52.407 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.558+0000 7f1bda050640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd41058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:52.407 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.558+0000 7f1bda050640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd4105eb0 con 0x7f1bd41054c0 2024-09-16T08:53:52.407 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.558+0000 7f1bd37fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd41058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:52.408 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.558+0000 7f1bd37fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd41058c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46996/0 (socket says 172.21.15.111:46996) 2024-09-16T08:53:52.408 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.558+0000 7f1bd37fe640 1 -- 172.21.15.111:0/1631767871 learned_addr learned my addr 172.21.15.111:0/1631767871 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:52.408 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.558+0000 7f1bd37fe640 1 -- 172.21.15.111:0/1631767871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd4106030 con 0x7f1bd41054c0 2024-09-16T08:53:52.408 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.559+0000 7f1bd37fe640 1 --2- 172.21.15.111:0/1631767871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd41058c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009820 tx=0x7f1bc402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8183cae160ec186e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:52.408 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.559+0000 7f1bd2ffd640 1 -- 172.21.15.111:0/1631767871 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bc4037070 con 0x7f1bd41054c0 2024-09-16T08:53:52.408 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.559+0000 7f1bd2ffd640 1 -- 172.21.15.111:0/1631767871 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f1bc4032870 con 0x7f1bd41054c0 2024-09-16T08:53:52.408 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.559+0000 7f1bd2ffd640 1 -- 172.21.15.111:0/1631767871 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bc40305d0 con 0x7f1bd41054c0 2024-09-16T08:53:52.408 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.560+0000 7f1bda050640 1 -- 172.21.15.111:0/1631767871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 msgr2=0x7f1bd41058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:52.408 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.560+0000 7f1bda050640 1 --2- 172.21.15.111:0/1631767871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd41058c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009820 tx=0x7f1bc402a270 comp rx=0 tx=0).stop 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.560+0000 7f1bda050640 1 -- 172.21.15.111:0/1631767871 shutdown_connections 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.560+0000 7f1bda050640 1 --2- 172.21.15.111:0/1631767871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd41058c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.560+0000 7f1bda050640 1 -- 172.21.15.111:0/1631767871 >> 172.21.15.111:0/1631767871 conn(0x7f1bd4100df0 msgr2=0x7f1bd4103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.560+0000 7f1bda050640 1 -- 172.21.15.111:0/1631767871 shutdown_connections 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.560+0000 7f1bda050640 1 -- 172.21.15.111:0/1631767871 wait complete. 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.560+0000 7f1bda050640 1 Processor -- start 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.560+0000 7f1bda050640 1 -- start start 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bda050640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bda050640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd4109940 con 0x7f1bd41054c0 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bd37fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bd37fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd419a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47002/0 (socket says 172.21.15.111:47002) 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bd37fe640 1 -- 172.21.15.111:0/721519945 learned_addr learned my addr 172.21.15.111:0/721519945 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:52.409 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bd37fe640 1 -- 172.21.15.111:0/721519945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd419ac20 con 0x7f1bd41054c0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bd37fe640 1 --2- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd419a6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40090b0 tx=0x7f1bc4030b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bd17fa640 1 -- 172.21.15.111:0/721519945 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bc403f020 con 0x7f1bd41054c0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bd17fa640 1 -- 172.21.15.111:0/721519945 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f1bc4030e10 con 0x7f1bd41054c0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd419aeb0 con 0x7f1bd41054c0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bd17fa640 1 -- 172.21.15.111:0/721519945 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bc4037040 con 0x7f1bd41054c0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.561+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd419b870 con 0x7f1bd41054c0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.562+0000 7f1bd17fa640 1 -- 172.21.15.111:0/721519945 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7f1bc4042020 con 0x7f1bd41054c0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.562+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd4109ac0 con 0x7f1bd41054c0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.562+0000 7f1bd17fa640 1 --2- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f1bc003f650 0x7f1bc0041b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.563+0000 7f1bd17fa640 1 -- 172.21.15.111:0/721519945 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1bc4074660 con 0x7f1bd41054c0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.563+0000 7f1bcaffd640 1 --2- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f1bc003f650 0x7f1bc0041b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.563+0000 7f1bcaffd640 1 --2- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f1bc003f650 0x7f1bc0041b10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc0098c0 tx=0x7f1bbc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.567+0000 7f1bd17fa640 1 -- 172.21.15.111:0/721519945 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f1bc4049020 con 0x7f1bd41054c0 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:48.746+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi111", "addr": "172.21.15.111", "target": ["mon-mgr", ""]}) -- 0x7f1bd419c0b0 con 0x7f1bc003f650 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.783+0000 7f1bd17fa640 1 -- 172.21.15.111:0/721519945 <== mgr.14118 v2:172.21.15.111:6800/2366462893 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f1bd419c0b0 con 0x7f1bc003f650 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f1bc003f650 msgr2=0x7f1bc0041b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 --2- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f1bc003f650 0x7f1bc0041b10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc0098c0 tx=0x7f1bbc006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 msgr2=0x7f1bd419a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 --2- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd419a6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40090b0 tx=0x7f1bc4030b50 comp rx=0 tx=0).stop 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 shutdown_connections 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 --2- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f1bc003f650 0x7f1bc0041b10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 --2- 172.21.15.111:0/721519945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd41054c0 0x7f1bd419a6c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 >> 172.21.15.111:0/721519945 conn(0x7f1bd4100df0 msgr2=0x7f1bd4078f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 shutdown_connections 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:51.786+0000 7f1bda050640 1 -- 172.21.15.111:0/721519945 wait complete. 2024-09-16T08:53:52.411 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying unmanaged mon service... 2024-09-16T08:53:53.087 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:52 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:53.087 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:52 smithi111 ceph-mon[29803]: Added host smithi111 2024-09-16T08:53:53.087 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:52 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:53:53.874 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-16T08:53:53.874 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.859+0000 7f170ad5c640 1 Processor -- start 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.859+0000 7f170ad5c640 1 -- start start 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.860+0000 7f170ad5c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170407a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.860+0000 7f170ad5c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f170407c0d0 con 0x7f170407bc20 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.860+0000 7f1708af8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170407a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.860+0000 7f1708af8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170407a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52674/0 (socket says 172.21.15.111:52674) 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.860+0000 7f1708af8640 1 -- 172.21.15.111:0/4059755768 learned_addr learned my addr 172.21.15.111:0/4059755768 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.860+0000 7f1708af8640 1 -- 172.21.15.111:0/4059755768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170407a7c0 con 0x7f170407bc20 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f1708af8640 1 --2- 172.21.15.111:0/4059755768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170407a280 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009820 tx=0x7f16f802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d044025c80e16a5b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f17037fe640 1 -- 172.21.15.111:0/4059755768 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16f8037070 con 0x7f170407bc20 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f17037fe640 1 -- 172.21.15.111:0/4059755768 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f16f8032870 con 0x7f170407bc20 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f17037fe640 1 -- 172.21.15.111:0/4059755768 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16f80305d0 con 0x7f170407bc20 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f170ad5c640 1 -- 172.21.15.111:0/4059755768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 msgr2=0x7f170407a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:53.875 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f170ad5c640 1 --2- 172.21.15.111:0/4059755768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170407a280 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009820 tx=0x7f16f802a270 comp rx=0 tx=0).stop 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f170ad5c640 1 -- 172.21.15.111:0/4059755768 shutdown_connections 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f170ad5c640 1 --2- 172.21.15.111:0/4059755768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170407a280 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f170ad5c640 1 -- 172.21.15.111:0/4059755768 >> 172.21.15.111:0/4059755768 conn(0x7f1704101ea0 msgr2=0x7f1704104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.861+0000 7f170ad5c640 1 -- 172.21.15.111:0/4059755768 shutdown_connections 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.862+0000 7f170ad5c640 1 -- 172.21.15.111:0/4059755768 wait complete. 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.862+0000 7f170ad5c640 1 Processor -- start 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.862+0000 7f170ad5c640 1 -- start start 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.862+0000 7f170ad5c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.862+0000 7f170ad5c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f170410cb50 con 0x7f170407bc20 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.862+0000 7f1708af8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.862+0000 7f1708af8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170419c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52684/0 (socket says 172.21.15.111:52684) 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.862+0000 7f1708af8640 1 -- 172.21.15.111:0/49605611 learned_addr learned my addr 172.21.15.111:0/49605611 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.863+0000 7f1708af8640 1 -- 172.21.15.111:0/49605611 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170419ce50 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.863+0000 7f1708af8640 1 --2- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170419c8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f16f80090b0 tx=0x7f16f8030b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.863+0000 7f1701ffb640 1 -- 172.21.15.111:0/49605611 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16f803f020 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.863+0000 7f1701ffb640 1 -- 172.21.15.111:0/49605611 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 972+0+0 (secure 0 0 0) 0x7f16f8030e10 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.863+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170419d0e0 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.863+0000 7f1701ffb640 1 -- 172.21.15.111:0/49605611 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16f8037040 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.863+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170419daa0 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.864+0000 7f1701ffb640 1 -- 172.21.15.111:0/49605611 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7f16f8042020 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.864+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f170407a300 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.864+0000 7f1701ffb640 1 --2- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f16f403f650 0x7f16f4041b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.864+0000 7f1701ffb640 1 -- 172.21.15.111:0/49605611 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f16f80745e0 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.864+0000 7f1703fff640 1 --2- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f16f403f650 0x7f16f4041b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.865+0000 7f1703fff640 1 --2- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f16f403f650 0x7f16f4041b10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f16f00098c0 tx=0x7f16f0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:52.869+0000 7f1701ffb640 1 -- 172.21.15.111:0/49605611 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f16f8049020 con 0x7f170407bc20 2024-09-16T08:53:53.877 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.085+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f1704103290 con 0x7f16f403f650 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.089+0000 7f1701ffb640 1 -- 172.21.15.111:0/49605611 <== mgr.14118 v2:172.21.15.111:6800/2366462893 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1704103290 con 0x7f16f403f650 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f16f403f650 msgr2=0x7f16f4041b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 --2- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f16f403f650 0x7f16f4041b10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f16f00098c0 tx=0x7f16f0006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 msgr2=0x7f170419c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 --2- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170419c8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f16f80090b0 tx=0x7f16f8030b50 comp rx=0 tx=0).stop 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 shutdown_connections 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 --2- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f16f403f650 0x7f16f4041b10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 --2- 172.21.15.111:0/49605611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f170407bc20 0x7f170419c8f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 >> 172.21.15.111:0/49605611 conn(0x7f1704101ea0 msgr2=0x7f1704102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 shutdown_connections 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:53.091+0000 7f170ad5c640 1 -- 172.21.15.111:0/49605611 wait complete. 2024-09-16T08:53:53.878 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying unmanaged mgr service... 2024-09-16T08:53:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:54 smithi111 ceph-mon[29803]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:53:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:54 smithi111 ceph-mon[29803]: Saving service mon spec with placement count:5 2024-09-16T08:53:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:54 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:56.305 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:56 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:56.305 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:56 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:56.305 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:56 smithi111 ceph-mon[29803]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:53:56.305 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:56 smithi111 ceph-mon[29803]: Saving service mgr spec with placement count:2 2024-09-16T08:53:56.305 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:56 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.564+0000 7f4ac7d91640 1 Processor -- start 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.565+0000 7f4ac7d91640 1 -- start start 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.565+0000 7f4ac7d91640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac01058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.565+0000 7f4ac7d91640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac0105eb0 con 0x7f4ac01054c0 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.565+0000 7f4ac5b2d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac01058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.565+0000 7f4ac5b2d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac01058c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52686/0 (socket says 172.21.15.111:52686) 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.565+0000 7f4ac5b2d640 1 -- 172.21.15.111:0/2359622797 learned_addr learned my addr 172.21.15.111:0/2359622797 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.565+0000 7f4ac5b2d640 1 -- 172.21.15.111:0/2359622797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac0106030 con 0x7f4ac01054c0 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.566+0000 7f4ac5b2d640 1 --2- 172.21.15.111:0/2359622797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac01058c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009820 tx=0x7f4abc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d99dc002a18299 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.566+0000 7f4ac4b2b640 1 -- 172.21.15.111:0/2359622797 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4abc037070 con 0x7f4ac01054c0 2024-09-16T08:53:56.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.566+0000 7f4ac4b2b640 1 -- 172.21.15.111:0/2359622797 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f4abc032890 con 0x7f4ac01054c0 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.566+0000 7f4ac4b2b640 1 -- 172.21.15.111:0/2359622797 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4abc030550 con 0x7f4ac01054c0 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.567+0000 7f4ac7d91640 1 -- 172.21.15.111:0/2359622797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 msgr2=0x7f4ac01058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.567+0000 7f4ac7d91640 1 --2- 172.21.15.111:0/2359622797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac01058c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009820 tx=0x7f4abc02a270 comp rx=0 tx=0).stop 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.567+0000 7f4ac7d91640 1 -- 172.21.15.111:0/2359622797 shutdown_connections 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.567+0000 7f4ac7d91640 1 --2- 172.21.15.111:0/2359622797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac01058c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.567+0000 7f4ac7d91640 1 -- 172.21.15.111:0/2359622797 >> 172.21.15.111:0/2359622797 conn(0x7f4ac0100df0 msgr2=0x7f4ac0103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.567+0000 7f4ac7d91640 1 -- 172.21.15.111:0/2359622797 shutdown_connections 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.567+0000 7f4ac7d91640 1 -- 172.21.15.111:0/2359622797 wait complete. 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.568+0000 7f4ac7d91640 1 Processor -- start 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.568+0000 7f4ac7d91640 1 -- start start 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.568+0000 7f4ac7d91640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac0199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.568+0000 7f4ac7d91640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac0109940 con 0x7f4ac01054c0 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.568+0000 7f4ac5b2d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac0199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.568+0000 7f4ac5b2d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac0199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52702/0 (socket says 172.21.15.111:52702) 2024-09-16T08:53:56.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.568+0000 7f4ac5b2d640 1 -- 172.21.15.111:0/970341888 learned_addr learned my addr 172.21.15.111:0/970341888 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.569+0000 7f4ac5b2d640 1 -- 172.21.15.111:0/970341888 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac0199ab0 con 0x7f4ac01054c0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.569+0000 7f4ac5b2d640 1 --2- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac0199550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4abc032860 tx=0x7f4abc030a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.569+0000 7f4ab6ffd640 1 -- 172.21.15.111:0/970341888 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4abc037040 con 0x7f4ac01054c0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.569+0000 7f4ab6ffd640 1 -- 172.21.15.111:0/970341888 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f4abc040070 con 0x7f4ac01054c0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.569+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac0199d40 con 0x7f4ac01054c0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.569+0000 7f4ab6ffd640 1 -- 172.21.15.111:0/970341888 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4abc03bcc0 con 0x7f4ac01054c0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.569+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac019a700 con 0x7f4ac01054c0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.570+0000 7f4ab6ffd640 1 -- 172.21.15.111:0/970341888 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7f4abc045460 con 0x7f4ac01054c0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.570+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac0109ac0 con 0x7f4ac01054c0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.570+0000 7f4ab6ffd640 1 --2- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f4aac03f6a0 0x7f4aac041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.570+0000 7f4ab6ffd640 1 -- 172.21.15.111:0/970341888 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4abc0741a0 con 0x7f4ac01054c0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.570+0000 7f4ac532c640 1 --2- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f4aac03f6a0 0x7f4aac041b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.571+0000 7f4ac532c640 1 --2- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f4aac03f6a0 0x7f4aac041b60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00098c0 tx=0x7f4ab0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:56.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.575+0000 7f4ab6ffd640 1 -- 172.21.15.111:0/970341888 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f4abc036170 con 0x7f4ac01054c0 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.787+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4ac01021e0 con 0x7f4aac03f6a0 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.791+0000 7f4ab6ffd640 1 -- 172.21.15.111:0/970341888 <== mgr.14118 v2:172.21.15.111:6800/2366462893 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4ac01021e0 con 0x7f4aac03f6a0 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.793+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f4aac03f6a0 msgr2=0x7f4aac041b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.793+0000 7f4ac7d91640 1 --2- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f4aac03f6a0 0x7f4aac041b60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00098c0 tx=0x7f4ab0006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.793+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 msgr2=0x7f4ac0199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.793+0000 7f4ac7d91640 1 --2- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac0199550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4abc032860 tx=0x7f4abc030a80 comp rx=0 tx=0).stop 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.794+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 shutdown_connections 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.794+0000 7f4ac7d91640 1 --2- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f4aac03f6a0 0x7f4aac041b60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.794+0000 7f4ac7d91640 1 --2- 172.21.15.111:0/970341888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ac01054c0 0x7f4ac0199550 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.794+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 >> 172.21.15.111:0/970341888 conn(0x7f4ac0100df0 msgr2=0x7f4ac0101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.794+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 shutdown_connections 2024-09-16T08:53:56.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:55.794+0000 7f4ac7d91640 1 -- 172.21.15.111:0/970341888 wait complete. 2024-09-16T08:53:58.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.367+0000 7f2242cd0640 1 Processor -- start 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.367+0000 7f2242cd0640 1 -- start start 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.367+0000 7f2242cd0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c105e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.367+0000 7f2242cd0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1063f0 con 0x7f223c105a20 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.367+0000 7f2240a6c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c105e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.367+0000 7f2240a6c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c105e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52714/0 (socket says 172.21.15.111:52714) 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.367+0000 7f2240a6c640 1 -- 172.21.15.111:0/225957552 learned_addr learned my addr 172.21.15.111:0/225957552 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.368+0000 7f2240a6c640 1 -- 172.21.15.111:0/225957552 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223c106570 con 0x7f223c105a20 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.368+0000 7f2240a6c640 1 --2- 172.21.15.111:0/225957552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c105e20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2230009820 tx=0x7f223002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13599dc4f3a3bf72 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.368+0000 7f223b7fe640 1 -- 172.21.15.111:0/225957552 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2230037070 con 0x7f223c105a20 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.368+0000 7f223b7fe640 1 -- 172.21.15.111:0/225957552 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f2230032890 con 0x7f223c105a20 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.368+0000 7f223b7fe640 1 -- 172.21.15.111:0/225957552 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2230030550 con 0x7f223c105a20 2024-09-16T08:53:58.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.369+0000 7f2242cd0640 1 -- 172.21.15.111:0/225957552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 msgr2=0x7f223c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.369+0000 7f2242cd0640 1 --2- 172.21.15.111:0/225957552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c105e20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2230009820 tx=0x7f223002a270 comp rx=0 tx=0).stop 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.369+0000 7f2242cd0640 1 -- 172.21.15.111:0/225957552 shutdown_connections 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.369+0000 7f2242cd0640 1 --2- 172.21.15.111:0/225957552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c105e20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.369+0000 7f2242cd0640 1 -- 172.21.15.111:0/225957552 >> 172.21.15.111:0/225957552 conn(0x7f223c100560 msgr2=0x7f223c1029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.369+0000 7f2242cd0640 1 -- 172.21.15.111:0/225957552 shutdown_connections 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.369+0000 7f2242cd0640 1 -- 172.21.15.111:0/225957552 wait complete. 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2242cd0640 1 Processor -- start 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2242cd0640 1 -- start start 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2242cd0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2242cd0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f223c107620 con 0x7f223c105a20 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2240a6c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2240a6c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52720/0 (socket says 172.21.15.111:52720) 2024-09-16T08:53:58.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2240a6c640 1 -- 172.21.15.111:0/3620490145 learned_addr learned my addr 172.21.15.111:0/3620490145 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2240a6c640 1 -- 172.21.15.111:0/3620490145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223c19ab70 con 0x7f223c105a20 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2240a6c640 1 --2- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c19a610 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2230030ad0 tx=0x7f2230030b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.370+0000 7f2239ffb640 1 -- 172.21.15.111:0/3620490145 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2230037040 con 0x7f223c105a20 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.371+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223c19ae00 con 0x7f223c105a20 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.371+0000 7f2239ffb640 1 -- 172.21.15.111:0/3620490145 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f2230040070 con 0x7f223c105a20 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.371+0000 7f2239ffb640 1 -- 172.21.15.111:0/3620490145 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f223003a390 con 0x7f223c105a20 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.371+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223c19b7c0 con 0x7f223c105a20 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.372+0000 7f2239ffb640 1 -- 172.21.15.111:0/3620490145 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7f223003b430 con 0x7f223c105a20 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.372+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f223c105e20 con 0x7f223c105a20 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.372+0000 7f2239ffb640 1 --2- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f222c03f650 0x7f222c041b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.372+0000 7f2239ffb640 1 -- 172.21.15.111:0/3620490145 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2230074210 con 0x7f223c105a20 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.372+0000 7f223bfff640 1 --2- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f222c03f650 0x7f222c041b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.372+0000 7f223bfff640 1 --2- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f222c03f650 0x7f222c041b10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f22280098c0 tx=0x7f2228006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:58.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.376+0000 7f2239ffb640 1 -- 172.21.15.111:0/3620490145 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f223003b720 con 0x7f223c105a20 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.559+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f223c19bf00 con 0x7f223c105a20 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.561+0000 7f2239ffb640 1 -- 172.21.15.111:0/3620490145 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f223003abb0 con 0x7f223c105a20 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f222c03f650 msgr2=0x7f222c041b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 --2- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f222c03f650 0x7f222c041b10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f22280098c0 tx=0x7f2228006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 msgr2=0x7f223c19a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 --2- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c19a610 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2230030ad0 tx=0x7f2230030b00 comp rx=0 tx=0).stop 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 shutdown_connections 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 --2- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f222c03f650 0x7f222c041b10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 --2- 172.21.15.111:0/3620490145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105a20 0x7f223c19a610 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 >> 172.21.15.111:0/3620490145 conn(0x7f223c100560 msgr2=0x7f223c101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 shutdown_connections 2024-09-16T08:53:58.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:57.564+0000 7f2242cd0640 1 -- 172.21.15.111:0/3620490145 wait complete. 2024-09-16T08:53:58.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:58 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3620490145' entity='client.admin' 2024-09-16T08:53:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:58 smithi111 ceph-mon[29803]: from='mgr.14118 172.21.15.111:0/1205560712' entity='mgr.y' 2024-09-16T08:53:59.463 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.575+0000 7ff1ae5a7640 1 Processor -- start 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.575+0000 7ff1ae5a7640 1 -- start start 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.575+0000 7ff1ae5a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.575+0000 7ff1ae5a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a8109710 con 0x7ff1a807a450 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.575+0000 7ff1a7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.575+0000 7ff1a7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52734/0 (socket says 172.21.15.111:52734) 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.575+0000 7ff1a7fff640 1 -- 172.21.15.111:0/4162051548 learned_addr learned my addr 172.21.15.111:0/4162051548 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.575+0000 7ff1a7fff640 1 -- 172.21.15.111:0/4162051548 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a8109890 con 0x7ff1a807a450 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.576+0000 7ff1a7fff640 1 --2- 172.21.15.111:0/4162051548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a81091d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff198009820 tx=0x7ff19802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269dabd758835c0e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.576+0000 7ff1a6ffd640 1 -- 172.21.15.111:0/4162051548 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff198037070 con 0x7ff1a807a450 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.576+0000 7ff1a6ffd640 1 -- 172.21.15.111:0/4162051548 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7ff198032890 con 0x7ff1a807a450 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.576+0000 7ff1a6ffd640 1 -- 172.21.15.111:0/4162051548 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1980305d0 con 0x7ff1a807a450 2024-09-16T08:53:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.577+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/4162051548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 msgr2=0x7ff1a81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.577+0000 7ff1ae5a7640 1 --2- 172.21.15.111:0/4162051548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a81091d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff198009820 tx=0x7ff19802a300 comp rx=0 tx=0).stop 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.577+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/4162051548 shutdown_connections 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.577+0000 7ff1ae5a7640 1 --2- 172.21.15.111:0/4162051548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a81091d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.577+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/4162051548 >> 172.21.15.111:0/4162051548 conn(0x7ff1a8100e00 msgr2=0x7ff1a8103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.577+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/4162051548 shutdown_connections 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.577+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/4162051548 wait complete. 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.577+0000 7ff1ae5a7640 1 Processor -- start 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.577+0000 7ff1ae5a7640 1 -- start start 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1ae5a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a8106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1ae5a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a810a940 con 0x7ff1a807a450 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1a7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a8106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1a7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a8106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52738/0 (socket says 172.21.15.111:52738) 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1a7fff640 1 -- 172.21.15.111:0/1780675505 learned_addr learned my addr 172.21.15.111:0/1780675505 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:53:59.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1a7fff640 1 -- 172.21.15.111:0/1780675505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a8106d80 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1a7fff640 1 --2- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a8106820 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff19802a8c0 tx=0x7ff198030ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1a57fa640 1 -- 172.21.15.111:0/1780675505 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff198037040 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1a57fa640 1 -- 172.21.15.111:0/1780675505 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7ff198040070 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a8105170 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1a57fa640 1 -- 172.21.15.111:0/1780675505 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff19803bc40 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.578+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a81056f0 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.579+0000 7ff1a57fa640 1 -- 172.21.15.111:0/1780675505 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7ff19803b430 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.579+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a8107240 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.579+0000 7ff1a57fa640 1 --2- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff19003f650 0x7ff190041b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.579+0000 7ff1a57fa640 1 -- 172.21.15.111:0/1780675505 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff1980740d0 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.579+0000 7ff1a77fe640 1 --2- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff19003f650 0x7ff190041b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.580+0000 7ff1a77fe640 1 --2- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff19003f650 0x7ff190041b10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0098c0 tx=0x7ff19c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.584+0000 7ff1a57fa640 1 -- 172.21.15.111:0/1780675505 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7ff1980313a0 con 0x7ff1a807a450 2024-09-16T08:53:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.768+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7ff1a8105c90 con 0x7ff1a807a450 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.771+0000 7ff1a57fa640 1 -- 172.21.15.111:0/1780675505 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7ff19803ae00 con 0x7ff1a807a450 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.773+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff19003f650 msgr2=0x7ff190041b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.773+0000 7ff1ae5a7640 1 --2- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff19003f650 0x7ff190041b10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0098c0 tx=0x7ff19c006db0 comp rx=0 tx=0).stop 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.773+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 msgr2=0x7ff1a8106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.773+0000 7ff1ae5a7640 1 --2- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a8106820 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff19802a8c0 tx=0x7ff198030ab0 comp rx=0 tx=0).stop 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.774+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 shutdown_connections 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.774+0000 7ff1ae5a7640 1 --2- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7ff19003f650 0x7ff190041b10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.774+0000 7ff1ae5a7640 1 --2- 172.21.15.111:0/1780675505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1a807a450 0x7ff1a8106820 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.774+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 >> 172.21.15.111:0/1780675505 conn(0x7ff1a8100e00 msgr2=0x7ff1a81019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.774+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 shutdown_connections 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:53:58.774+0000 7ff1ae5a7640 1 -- 172.21.15.111:0/1780675505 wait complete. 2024-09-16T08:53:59.467 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling the dashboard module... 2024-09-16T08:54:00.112 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:53:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1780675505' entity='client.admin' 2024-09-16T08:54:01.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:00 smithi111 ceph-mon[29803]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:02.092 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3813758231' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-16T08:54:02.092 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:01 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ignoring --setuser ceph since I am not root 2024-09-16T08:54:02.092 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:01 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ignoring --setgroup ceph since I am not root 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.652+0000 7fcf91065640 1 Processor -- start 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.652+0000 7fcf91065640 1 -- start start 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.652+0000 7fcf91065640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c105ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.652+0000 7fcf91065640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c1063e0 con 0x7fcf8c105aa0 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.652+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c105ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.652+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c105ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38772/0 (socket says 172.21.15.111:38772) 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.652+0000 7fcf8ad9d640 1 -- 172.21.15.111:0/355535330 learned_addr learned my addr 172.21.15.111:0/355535330 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.653+0000 7fcf8ad9d640 1 -- 172.21.15.111:0/355535330 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8c106560 con 0x7fcf8c105aa0 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.653+0000 7fcf8ad9d640 1 --2- 172.21.15.111:0/355535330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c105ea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009820 tx=0x7fcf7c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c9d6b5773abc24 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:02.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.653+0000 7fcf89d9b640 1 -- 172.21.15.111:0/355535330 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf7c037070 con 0x7fcf8c105aa0 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.653+0000 7fcf89d9b640 1 -- 172.21.15.111:0/355535330 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fcf7c032890 con 0x7fcf8c105aa0 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.653+0000 7fcf89d9b640 1 -- 172.21.15.111:0/355535330 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf7c030550 con 0x7fcf8c105aa0 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.654+0000 7fcf91065640 1 -- 172.21.15.111:0/355535330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 msgr2=0x7fcf8c105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.654+0000 7fcf91065640 1 --2- 172.21.15.111:0/355535330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c105ea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009820 tx=0x7fcf7c02a270 comp rx=0 tx=0).stop 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.654+0000 7fcf91065640 1 -- 172.21.15.111:0/355535330 shutdown_connections 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.654+0000 7fcf91065640 1 --2- 172.21.15.111:0/355535330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c105ea0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.654+0000 7fcf91065640 1 -- 172.21.15.111:0/355535330 >> 172.21.15.111:0/355535330 conn(0x7fcf8c100560 msgr2=0x7fcf8c1029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.654+0000 7fcf91065640 1 -- 172.21.15.111:0/355535330 shutdown_connections 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.654+0000 7fcf91065640 1 -- 172.21.15.111:0/355535330 wait complete. 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.655+0000 7fcf91065640 1 Processor -- start 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.655+0000 7fcf91065640 1 -- start start 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.655+0000 7fcf91065640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:02.385 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.655+0000 7fcf91065640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c107610 con 0x7fcf8c105aa0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.655+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.655+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c19a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38776/0 (socket says 172.21.15.111:38776) 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.655+0000 7fcf8ad9d640 1 -- 172.21.15.111:0/3813758231 learned_addr learned my addr 172.21.15.111:0/3813758231 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.656+0000 7fcf8ad9d640 1 -- 172.21.15.111:0/3813758231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8c19ab00 con 0x7fcf8c105aa0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.656+0000 7fcf8ad9d640 1 --2- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c19a5a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c032860 tx=0x7fcf7c030a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.656+0000 7fcf73fff640 1 -- 172.21.15.111:0/3813758231 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf7c037040 con 0x7fcf8c105aa0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.656+0000 7fcf73fff640 1 -- 172.21.15.111:0/3813758231 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fcf7c040070 con 0x7fcf8c105aa0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.656+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf8c19ad90 con 0x7fcf8c105aa0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.656+0000 7fcf73fff640 1 -- 172.21.15.111:0/3813758231 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf7c03bd60 con 0x7fcf8c105aa0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.656+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf8c19b750 con 0x7fcf8c105aa0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.657+0000 7fcf73fff640 1 -- 172.21.15.111:0/3813758231 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) ==== 51765+0+0 (secure 0 0 0) 0x7fcf7c045460 con 0x7fcf8c105aa0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.657+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf8c105ea0 con 0x7fcf8c105aa0 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.657+0000 7fcf73fff640 1 --2- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fcf7403f650 0x7fcf74041b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:02.386 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.657+0000 7fcf73fff640 1 -- 172.21.15.111:0/3813758231 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcf7c074210 con 0x7fcf8c105aa0 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.658+0000 7fcf8a59c640 1 --2- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fcf7403f650 0x7fcf74041b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.658+0000 7fcf8a59c640 1 --2- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fcf7403f650 0x7fcf74041b10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcf800098c0 tx=0x7fcf80006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.662+0000 7fcf73fff640 1 -- 172.21.15.111:0/3813758231 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fcf7c030dc0 con 0x7fcf8c105aa0 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:00.892+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fcf8c19bf50 con 0x7fcf8c105aa0 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.776+0000 7fcf73fff640 1 -- 172.21.15.111:0/3813758231 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fcf7c040af0 con 0x7fcf8c105aa0 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.778+0000 7fcf73fff640 1 -- 172.21.15.111:0/3813758231 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 9) ==== 51778+0+0 (secure 0 0 0) 0x7fcf7c043c80 con 0x7fcf8c105aa0 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.779+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fcf7403f650 msgr2=0x7fcf74041b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.779+0000 7fcf91065640 1 --2- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fcf7403f650 0x7fcf74041b10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcf800098c0 tx=0x7fcf80006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.779+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 msgr2=0x7fcf8c19a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.779+0000 7fcf91065640 1 --2- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c19a5a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c032860 tx=0x7fcf7c030a80 comp rx=0 tx=0).stop 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.779+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 shutdown_connections 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.779+0000 7fcf91065640 1 --2- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7fcf7403f650 0x7fcf74041b10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.779+0000 7fcf91065640 1 --2- 172.21.15.111:0/3813758231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8c105aa0 0x7fcf8c19a5a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:02.387 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.779+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 >> 172.21.15.111:0/3813758231 conn(0x7fcf8c100560 msgr2=0x7fcf8c1047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:02.388 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.780+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 shutdown_connections 2024-09-16T08:54:02.388 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:01.780+0000 7fcf91065640 1 -- 172.21.15.111:0/3813758231 wait complete. 2024-09-16T08:54:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3813758231' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-16T08:54:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:02 smithi111 ceph-mon[29803]: mgrmap e9: y(active, since 22s) 2024-09-16T08:54:03.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.843+0000 7f804dbdc640 1 Processor -- start 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.844+0000 7f804dbdc640 1 -- start start 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.844+0000 7f804dbdc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048109220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.844+0000 7f804dbdc640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8048109760 con 0x7f804807a380 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.844+0000 7f80477fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048109220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.844+0000 7f80477fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048109220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38806/0 (socket says 172.21.15.111:38806) 2024-09-16T08:54:03.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.844+0000 7f80477fe640 1 -- 172.21.15.111:0/484311161 learned_addr learned my addr 172.21.15.111:0/484311161 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.844+0000 7f80477fe640 1 -- 172.21.15.111:0/484311161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80481098e0 con 0x7f804807a380 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.845+0000 7f80477fe640 1 --2- 172.21.15.111:0/484311161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048109220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8038009820 tx=0x7f803802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d880f600932ccc36 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.845+0000 7f80467fc640 1 -- 172.21.15.111:0/484311161 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8038037070 con 0x7f804807a380 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.845+0000 7f80467fc640 1 -- 172.21.15.111:0/484311161 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f8038032890 con 0x7f804807a380 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.845+0000 7f80467fc640 1 -- 172.21.15.111:0/484311161 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8038030550 con 0x7f804807a380 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.846+0000 7f804dbdc640 1 -- 172.21.15.111:0/484311161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 msgr2=0x7f8048109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.846+0000 7f804dbdc640 1 --2- 172.21.15.111:0/484311161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048109220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8038009820 tx=0x7f803802a270 comp rx=0 tx=0).stop 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.846+0000 7f804dbdc640 1 -- 172.21.15.111:0/484311161 shutdown_connections 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.846+0000 7f804dbdc640 1 --2- 172.21.15.111:0/484311161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048109220 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.846+0000 7f804dbdc640 1 -- 172.21.15.111:0/484311161 >> 172.21.15.111:0/484311161 conn(0x7f8048100df0 msgr2=0x7f8048103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.846+0000 7f804dbdc640 1 -- 172.21.15.111:0/484311161 shutdown_connections 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.846+0000 7f804dbdc640 1 -- 172.21.15.111:0/484311161 wait complete. 2024-09-16T08:54:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.847+0000 7f804dbdc640 1 Processor -- start 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.847+0000 7f804dbdc640 1 -- start start 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.847+0000 7f804dbdc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048106870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.847+0000 7f804dbdc640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f804810a8d0 con 0x7f804807a380 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.847+0000 7f80477fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048106870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.847+0000 7f80477fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048106870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38822/0 (socket says 172.21.15.111:38822) 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.847+0000 7f80477fe640 1 -- 172.21.15.111:0/727664034 learned_addr learned my addr 172.21.15.111:0/727664034 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.847+0000 7f80477fe640 1 -- 172.21.15.111:0/727664034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8048106dd0 con 0x7f804807a380 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.847+0000 7f80477fe640 1 --2- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048106870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80380097f0 tx=0x7f8038030ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.848+0000 7f8044ff9640 1 -- 172.21.15.111:0/727664034 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8038037040 con 0x7f804807a380 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.848+0000 7f8044ff9640 1 -- 172.21.15.111:0/727664034 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f8038040070 con 0x7f804807a380 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.848+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80481051c0 con 0x7f804807a380 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.848+0000 7f8044ff9640 1 -- 172.21.15.111:0/727664034 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f803803bc90 con 0x7f804807a380 2024-09-16T08:54:03.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.848+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80481057a0 con 0x7f804807a380 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.849+0000 7f8044ff9640 1 -- 172.21.15.111:0/727664034 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) ==== 51778+0+0 (secure 0 0 0) 0x7f8038045460 con 0x7f804807a380 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.849+0000 7f8044ff9640 1 --2- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f803003f6a0 0x7f8030041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.849+0000 7f8046ffd640 1 -- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f803003f6a0 msgr2=0x7f8030041b60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2366462893 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.849+0000 7f8044ff9640 1 -- 172.21.15.111:0/727664034 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8038074200 con 0x7f804807a380 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.849+0000 7f8046ffd640 1 --2- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f803003f6a0 0x7f8030041b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.849+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8048106fa0 con 0x7f804807a380 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:02.854+0000 7f8044ff9640 1 -- 172.21.15.111:0/727664034 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f8038030c60 con 0x7f804807a380 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.050+0000 7f8046ffd640 1 -- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f803003f6a0 msgr2=0x7f8030041b60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2366462893 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.050+0000 7f8046ffd640 1 --2- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f803003f6a0 0x7f8030041b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.105+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f8048107170 con 0x7f804807a380 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.105+0000 7f8044ff9640 1 -- 172.21.15.111:0/727664034 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f80380443c0 con 0x7f804807a380 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.107+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f803003f6a0 msgr2=0x7f8030041b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.107+0000 7f804dbdc640 1 --2- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f803003f6a0 0x7f8030041b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:03.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.107+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 msgr2=0x7f8048106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:03.703 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.107+0000 7f804dbdc640 1 --2- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048106870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80380097f0 tx=0x7f8038030ad0 comp rx=0 tx=0).stop 2024-09-16T08:54:03.703 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.107+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 shutdown_connections 2024-09-16T08:54:03.703 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.107+0000 7f804dbdc640 1 --2- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f803003f6a0 0x7f8030041b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:03.703 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.107+0000 7f804dbdc640 1 --2- 172.21.15.111:0/727664034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f804807a380 0x7f8048106870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:03.703 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.107+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 >> 172.21.15.111:0/727664034 conn(0x7f8048100df0 msgr2=0x7f80481019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:03.703 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.108+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 shutdown_connections 2024-09-16T08:54:03.703 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:03.108+0000 7f804dbdc640 1 -- 172.21.15.111:0/727664034 wait complete. 2024-09-16T08:54:03.703 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for the mgr to restart... 2024-09-16T08:54:03.703 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr epoch 9... 2024-09-16T08:54:04.076 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/727664034' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-16T08:54:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: Active manager daemon y restarted 2024-09-16T08:54:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: Activating manager daemon y 2024-09-16T08:54:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: osdmap e3: 0 total, 0 up, 0 in 2024-09-16T08:54:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: mgrmap e10: y(active, starting, since 0.00328494s) 2024-09-16T08:54:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:54:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T08:54:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T08:54:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T08:54:07.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T08:54:07.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:06 smithi111 ceph-mon[29803]: Manager daemon y is now available 2024-09-16T08:54:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:07 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:07 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T08:54:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:07 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T08:54:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:07 smithi111 ceph-mon[29803]: mgrmap e11: y(active, since 1.00647s) 2024-09-16T08:54:08.574 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: 2024-09-16T08:54:08.381+0000 7ff8ea831640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:08:54:08] ENGINE Error in HTTPServer.serve 2024-09-16T08:54:08.574 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: Traceback (most recent call last): 2024-09-16T08:54:08.574 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T08:54:08.574 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self._connections.run(self.expiration_interval) 2024-09-16T08:54:08.574 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T08:54:08.574 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self._run(expiration_interval) 2024-09-16T08:54:08.574 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: s = self.context.wrap_socket( 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: return self.sslsocket_class._create( 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self.do_handshake() 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self._sslobj.do_handshake() 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T08:54:08.575 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: 2024-09-16T08:54:08.582 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-09-16T08:54:08.582 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.156+0000 7f49a4ef7640 1 Processor -- start 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.156+0000 7f49a4ef7640 1 -- start start 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.157+0000 7f49a4ef7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a010b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.157+0000 7f49a4ef7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f49a0079e30 con 0x7f49a0079a30 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.157+0000 7f499e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a010b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.157+0000 7f499e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a010b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38824/0 (socket says 172.21.15.111:38824) 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.157+0000 7f499e59c640 1 -- 172.21.15.111:0/1189938162 learned_addr learned my addr 172.21.15.111:0/1189938162 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.157+0000 7f499e59c640 1 -- 172.21.15.111:0/1189938162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a010bb40 con 0x7f49a0079a30 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.158+0000 7f499e59c640 1 --2- 172.21.15.111:0/1189938162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a010b600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4990009820 tx=0x7f499002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b115cda1d88a19aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:08.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.158+0000 7f499d59a640 1 -- 172.21.15.111:0/1189938162 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4990037070 con 0x7f49a0079a30 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.158+0000 7f499d59a640 1 -- 172.21.15.111:0/1189938162 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f4990032890 con 0x7f49a0079a30 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.158+0000 7f499d59a640 1 -- 172.21.15.111:0/1189938162 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4990030550 con 0x7f49a0079a30 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.158+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1189938162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 msgr2=0x7f49a010b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.158+0000 7f49a4ef7640 1 --2- 172.21.15.111:0/1189938162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a010b600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4990009820 tx=0x7f499002a270 comp rx=0 tx=0).stop 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.159+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1189938162 shutdown_connections 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.159+0000 7f49a4ef7640 1 --2- 172.21.15.111:0/1189938162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a010b600 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.159+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1189938162 >> 172.21.15.111:0/1189938162 conn(0x7f49a0100df0 msgr2=0x7f49a0103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.159+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1189938162 shutdown_connections 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.159+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1189938162 wait complete. 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.159+0000 7f49a4ef7640 1 Processor -- start 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.159+0000 7f49a4ef7640 1 -- start start 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f49a4ef7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:08.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f49a4ef7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f49a010cad0 con 0x7f49a0079a30 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f499e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f499e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a019c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38826/0 (socket says 172.21.15.111:38826) 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f499e59c640 1 -- 172.21.15.111:0/1147981035 learned_addr learned my addr 172.21.15.111:0/1147981035 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f499e59c640 1 -- 172.21.15.111:0/1147981035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a019cd20 con 0x7f49a0079a30 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f499e59c640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a019c7c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f49900097f0 tx=0x7f4990030ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4990037040 con 0x7f49a0079a30 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f4990040070 con 0x7f49a0079a30 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1147981035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a019cfb0 con 0x7f49a0079a30 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f499003bc90 con 0x7f49a0079a30 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.160+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1147981035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a019d9f0 con 0x7f49a0079a30 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.161+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) ==== 51778+0+0 (secure 0 0 0) 0x7f4990045460 con 0x7f49a0079a30 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.161+0000 7f49977fe640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 0x7f498c041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:08.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.161+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 --> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f498c042270 con 0x7f498c03f6a0 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.161+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4990074210 con 0x7f49a0079a30 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.161+0000 7f499dd9b640 1 -- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 msgr2=0x7f498c041b60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2366462893 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.161+0000 7f499dd9b640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 0x7f498c041b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.362+0000 7f499dd9b640 1 -- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 msgr2=0x7f498c041b60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2366462893 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.362+0000 7f499dd9b640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 0x7f498c041b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.763+0000 7f499dd9b640 1 -- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 msgr2=0x7f498c041b60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2366462893 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:04.763+0000 7f499dd9b640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 0x7f498c041b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:05.564+0000 7f499dd9b640 1 -- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 msgr2=0x7f498c041b60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2366462893 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:05.564+0000 7f499dd9b640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 0x7f498c041b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:06.871+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mgrmap(e 10) ==== 51545+0+0 (secure 0 0 0) 0x7f4990072cc0 con 0x7f49a0079a30 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:06.871+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 msgr2=0x7f498c041b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:06.871+0000 7f49977fe640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 0x7f498c041b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.873+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 11) ==== 51672+0+0 (secure 0 0 0) 0x7f499004bd10 con 0x7f49a0079a30 2024-09-16T08:54:08.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.874+0000 7f49977fe640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f498c042fe0 0x7f498c0453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.874+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4990046e50 con 0x7f498c042fe0 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.875+0000 7f499dd9b640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f498c042fe0 0x7f498c0453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.876+0000 7f499dd9b640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f498c042fe0 0x7f498c0453d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4988003980 tx=0x7f4988006e10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.876+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f4990046e50 con 0x7f498c042fe0 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.883+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1147981035 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f49a010b600 con 0x7f498c042fe0 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.883+0000 7f49977fe640 1 -- 172.21.15.111:0/1147981035 <== mgr.14152 v2:172.21.15.111:6800/3306957597 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f49a010b600 con 0x7f498c042fe0 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f498c042fe0 msgr2=0x7f498c0453d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f498c042fe0 0x7f498c0453d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4988003980 tx=0x7f4988006e10 comp rx=0 tx=0).stop 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 msgr2=0x7f49a019c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a019c7c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f49900097f0 tx=0x7f4990030ad0 comp rx=0 tx=0).stop 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1147981035 shutdown_connections 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f498c042fe0 0x7f498c0453d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:6800/2366462893,v1:172.21.15.111:6801/2366462893] conn(0x7f498c03f6a0 0x7f498c041b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 --2- 172.21.15.111:0/1147981035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a0079a30 0x7f49a019c7c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:08.588 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1147981035 >> 172.21.15.111:0/1147981035 conn(0x7f49a0100df0 msgr2=0x7f49a0101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:08.588 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1147981035 shutdown_connections 2024-09-16T08:54:08.588 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:07.884+0000 7f49a4ef7640 1 -- 172.21.15.111:0/1147981035 wait complete. 2024-09-16T08:54:08.588 INFO:teuthology.orchestra.run.smithi111.stdout:mgr epoch 9 is available 2024-09-16T08:54:08.588 INFO:teuthology.orchestra.run.smithi111.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-mon[29803]: [16/Sep/2024:08:54:08] ENGINE Bus STARTING 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-mon[29803]: [16/Sep/2024:08:54:08] ENGINE Serving on https://172.21.15.111:7150 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-mon[29803]: [16/Sep/2024:08:54:08] ENGINE Error in HTTPServer.serve 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout: Traceback (most recent call last): 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout: self._connections.run(self.expiration_interval) 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout: self._run(expiration_interval) 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T08:54:09.279 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: s = self.context.wrap_socket( 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: return self.sslsocket_class._create( 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: self.do_handshake() 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: self._sslobj.do_handshake() 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-mon[29803]: [16/Sep/2024:08:54:08] ENGINE Serving on http://172.21.15.111:8765 2024-09-16T08:54:09.280 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:08 smithi111 ceph-mon[29803]: [16/Sep/2024:08:54:08] ENGINE Bus STARTED 2024-09-16T08:54:10.229 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:09 smithi111 ceph-mon[29803]: mgrmap e12: y(active, since 2s) 2024-09-16T08:54:10.229 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:09 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:10.229 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:09 smithi111 ceph-mon[29803]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:10.229 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:09 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:11.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIFETCCAvmgAwIBAgIUIg6uql7+GWxOfvDNehHvNaFAmUkwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkxNjA4NTQxMFoXDTM0\nMDkxNzA4NTQxMFowGDEWMBQGA1UEAwwNMTcyLjIxLjE1LjExMTCCAiIwDQYJKoZI\nhvcNAQEBBQADggIPADCCAgoCggIBAJ890En4FNXbIEHIyKMhl2f8etDp+sFB+2rI\nmfT9qJWUYJuZ9gES5rWoLMD+T5KEw4o8ib8D0ca43pQoqz3WEfWEfyXgR76JXP9e\nAK82xZRjjswQGU4LyXcXpZwp0yPj2fVJ9G2qSsZ7LyOv1FYL50A2C4MrWv/0kXfb\nWTL8o4EW9LNX/XW6+N2khQ2Sr+4J3SuYB4J5pQGhGbhsvku0nMwhdDsAd0MrCa0r\nEB5Rw0N0jVqk+OCOjBc4PewmGDeaZLOmZPsVIciFi+6ROER7EkmM4jdFPqo9ICsp\n8NbW+TmdoaAdqjG30sfKSzGVFsA0LANvhqL4QfQiz3NWc6ZuUAmidRtMMfgjfufw\nIxuKO16YvxJKIVfBhhlAlhr/U2e+xdjqV3cSvE8fctdRTtMyCaHFMwU/ZHsf5TPz\niXUOnOqQd5SEvAKs/bqXYKZWc0P3PS+4bWzw8kCJInq7gAp0g4mTVNHM6TLv6Qeq\n8/u5h4e1MOoK96vtxSbNfUdnjJhKsENvccdJl4oQ7nC8TAC57GCVc+C8SL0yH/A0\ngzpHzmNtPZV9A1qAIb4URvcvEMfku6ja8xz4h5i20is+qK3cym0eoYQhsma4ceny\nj+2i+A1V14rxkX9gXfadyqaWWRSNbfPDldyRzvbOB1vYAR5KLi4wqnwhsYj3UVrr\nkevXe/AfAgMBAAGjVDBSMEIGA1UdEQQ7MDmCHnNtaXRoaTExMS5mcm9udC5zZXBp\nYS5jZXBoLmNvbYIRZGFzaGJvYXJkX3NlcnZlcnOHBKwVD28wDAYDVR0TAQH/BAIw\nADANBgkqhkiG9w0BAQsFAAOCAgEAZ2rMF0SHqhzsAvp1gXdjkYmfnmSLNUFUPOHr\nyOtAwJG6C8XBAHS3cBhNKEZI1NNI07eHse7x7Gp0Xlbq5NfkIDXKBBU5en2B/I95\nXqEmhDn4I4w2ikA+1i05Uqjxr8GOPXUAMxHaAB+a/J55B6ZRpaVjvILUhV2RgJ5k\nPt1/AyI66Y2ljR+q6AODHjVWSeTFiDBbz4lLNiDhVwkcuDw6p/V+hKaMWGt2zCra\nqPPYdakN8TDbon0vZjjRtRfAevM4s06Q7OfI7la7plgBGwd++Hu9XWK6hsNpS2ZQ\nu9MnKG7UXd6pZggp7XwQ1ULPxHvz/frLGR5hMFFlPizcqnX2jFEgH+niU7Ewg/Lo\nDrOZ3LghEvPim8jSPfaVSdY5Kaw9hAgMkKXHqeFIGchfDll2bxsNlnOrjKmWit5g\nLa3vsutYTZJLbZMEKHLVIIMv9piOwfmpGPQed10FVuaPMrfHSDzbPzMRklQR8gVc\nHfKIoMn5+4M0oh8i/JEURgrHd+U/1qHuKaCxuU/uNk+jbmy+RDyz26QDdPU9EI5M\nhfiu5QD26VeQHdyBMltRCyDemEodF5epeaPRmETwblGkNvwZRaTpo/YWVzPVA5AT\n2rvjwIw7bFqE0XKJ2Z9AWzkfh04OFhBL/nYc1/r3ntDBGFQGYRM523X5f2uoDHsb\nFwjw0ew=\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJJwIBAAKCAgEAnz3QSfgU1dsgQcjIoyGXZ/x60On6wUH7asiZ9P2olZRgm5n2\nARLmtagswP5PkoTDijyJvwPRxrjelCirPdYR9YR/JeBHvolc/14ArzbFlGOOzBAZ\nTgvJdxelnCnTI+PZ9Un0bapKxnsvI6/UVgvnQDYLgyta//SRd9tZMvyjgRb0s1f9\ndbr43aSFDZKv7gndK5gHgnmlAaEZuGy+S7SczCF0OwB3QysJrSsQHlHDQ3SNWqT4\n4I6MFzg97CYYN5pks6Zk+xUhyIWL7pE4RHsSSYziN0U+qj0gKynw1tb5OZ2hoB2q\nMbfSx8pLMZUWwDQsA2+GovhB9CLPc1Zzpm5QCaJ1G0wx+CN+5/AjG4o7Xpi/Ekoh\nV8GGGUCWGv9TZ77F2OpXdxK8Tx9y11FO0zIJocUzBT9kex/lM/OJdQ6c6pB3lIS8\nAqz9updgplZzQ/c9L7htbPDyQIkieruACnSDiZNU0czpMu/pB6rz+7mHh7Uw6gr3\nq+3FJs19R2eMmEqwQ29xx0mXihDucLxMALnsYJVz4LxIvTIf8DSDOkfOY209lX0D\nWoAhvhRG9y8Qx+S7qNrzHPiHmLbSKz6ordzKbR6hhCGyZrhx6fKP7aL4DVXXivGR\nf2Bd9p3KppZZFI1t88OV3JHO9s4HW9gBHkouLjCqfCGxiPdRWuuR69d78B8CAwEA\nAQKCAgAb7oREDUNeVa7dSyJCVwiPQV6rmXcWAgRfMSusKPgysJGXiZR4kL+uGn8r\nJRCBE0+XRlIlEr7vnArSsyWCVbPVvHVvB92dwstwsaTuFu3mPuZTzfkaUv/2J0wx\nzbZO488d8uhwef/eu8nDeTqGE1VvUmuHiQnPvq7rdqq5Qk5NvOCUOrlnNrIjcpGy\nFUpNtyY32NVsYfj7Ba6UxmJEtjwQpEeWd9jzi6+JU/XiU5YsFVwcC7bp8KcLBZ76\n9p2l1yuVxXiOACFqYl5urMNRY8ML6q3dYwSAaPNlDtjDXnbdz9QDWWC6ktNinw28\ndznD4gLcYWoc2O38rER2JzJm/Y+FMkhwo0qGU9kRk2TILSZKVqTXbwpc0BBHdLjh\nHaJiuQGwbU1prNNme/345m/XoNg37ZWZqMYuKMWvyrFf+UYS9Rglz07ZX+PGrWVs\n4trvjR7RJJ19eIJfi1GoxbQUQybOy+j4qWt5DTLa1ajewxOQFXjCvIPJTzDfopWN\nCpCH3FG2VOywAnAYLTdEbY2/eK0dnzSn+l1Zz3kHVpdy43ivnjSo+o8cOJ2YEkrO\noZlc/HEnMmvd0RjVH96Xj5o2YmCuAe4gTI4z/JKwldmy0OvF3AH5JMEhfSneLpsz\nMwE+D61UghTh6Cyxri9XDSvHU9KjA5sVOM17yHzOBsL/s4xrxQKCAQEAymhGXtuP\nUob/QHDhf62Sxt/IlSf3r7Xav4BEQuyr4tbHlwlZsjmVybbYN5LW3LbQxsYLAlgV\ncH8jDYS9nRk2aW2qLMYaqzm8UZDAl2fdE25tHdBq76ghDsWHdDn6wIm8QBBg8xCl\nRkESZbUAHEX1nNF3ZyQbBQei+gJ7m7rCFFNkCbjolm6FjP3ChUb3cptyTtDawRZG\n0g5l7jEXj/lEoVsLVq4POz1H3Hj6mFrIl+cBcYzc6roKPZ+U4qtIIsS6hmnFJ2/L\nGPL3wXTlmbjy4XY25iBRp//pP3CoQQ6YPsjC8Jvr3yJKPAqJayu+rk9KQCet0i0Q\nkfQq6OD5RtSrewKCAQEAyWejQsKR9lEZriP3kWtSuCK9okWvbcvBo37X8C2VBfsV\nZqOCCpTu/vCLQZSRKvs2gMTZDGdyqIJ7BuJ2VcEeOts33XDf9laDtU1vmZGclKGO\n5T8x/vobboiNV6rar2XYFnclpiTiDKsylA2zt9a6kcSO57FGWeB4jwH4O4iPdD+S\n2o8ghqQZXSuJLGeAKpvT71sqnT7c12/jsg4AlKI+Uklrx8CVJ5FH3csusoaYNRlu\nMxJELLLci8yMtlezaX532FFZ1hJy8QVZwXCIIMtq8Rf1MYcyopz6dlaoSkH3SvNx\nISf4wh3ASuzFf1UIrWLxB1Lw9vJeaHJGynLk5s/KrQKCAQBMsTX4uziTDJOq1xqa\nRipXMHPQEFTJPx6jcPZsZUC7sqXtlJ1n6mnXBHDonbrL5nk019VUDgwRlBq0BJfT\n7rmvON5CGM4CEt5ICZDvzWr28eBma5IZPJtJ/Y5Ve6KHUakCnTIe5y+4bu+Jmgtw\nqxFlWfjkG8ohHcOi903es+TVzsMY6/c1CKuRwJTz6u/XfQ7SzlAU2Vv8O3hjyDYl\nV02p1dV2An3C9HVGFJDfKYbYB6+EVtteFR/MOaswN7Yddm02gN0zFCjgSy+VHbXh\niTJ+O0yR5a0K+uxmHUmjXORS5b73bBzzbwqxtUxcTzbw1RpNNLun52tf4iRgGN5a\n+DoBAoIBAFsopHWQGX6ygHwYXXKdcAhV+R0ci2WXxuJYhCZ6iyv01F/re+2MMYyE\nlZMuSnJ419d6wX7ix7vLqyGXmNYBncwn8uYws74MtHp4PR32ETgY09OhQeDVlVr3\nRXo7o9WltCOe9BR6NzJivnbSqNh5wWI+5CusSlpdxlN8Ib3Ydo24RKDWDXPDjJNm\nJCVMB2/BrjB1RBl+HXkJdLbfrZWYxbp67uSvnxMUp8GUuGMnlnbaNs2449CTeE1y\nXPIfSd5i1p27XH0PRqUetdxfMc4eJB5RqLItLlb51EeBu5Q4vVRYUYkfTXGXA7Ul\n69JRS6cm1E9rVBOB6gc+ShpR3Jc3z9UCggEAPs7kgPfvRod69luymR2eqNK+0xjZ\nzUiqB68yry0xQEKzkAjZtj8nBzeULnDfkV/mN0E6fPHRm6j3VzVlOMT/Sqz73s/G\nHDbA/OWd+aMRJ9pOFMUD3KWH3n5gsRXCfuNOa1P2gCDITnvBZqoFLsrPjyte/eOm\nn8Z8OBoWE/LcY27eUU2GGqy9g/iYv2ulKDffOcIL2wBKvcPxBBbgUObqg92jBGG1\nrnJzaQ2jUxQXGqgPdEFakGPtktXnLlrBM38vtCNaw9BU3wIEZCWqFaLlVZIv2cOa\nSDzZdXbv+avWBhmSxW+38Lm3LRJllJCsPu/CJS9mRtGmtIRS6vlzU1V6/g==\n-----END RSA PRIVATE KEY-----\n"} 2024-09-16T08:54:11.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.361+0000 7f1e0751d640 1 Processor -- start 2024-09-16T08:54:11.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.362+0000 7f1e0751d640 1 -- start start 2024-09-16T08:54:11.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.362+0000 7f1e0751d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e001058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:11.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.362+0000 7f1e0751d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00105eb0 con 0x7f1e001054c0 2024-09-16T08:54:11.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.362+0000 7f1e052b9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e001058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:11.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.362+0000 7f1e052b9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e001058c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35778/0 (socket says 172.21.15.111:35778) 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.362+0000 7f1e052b9640 1 -- 172.21.15.111:0/2720925416 learned_addr learned my addr 172.21.15.111:0/2720925416 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.362+0000 7f1e052b9640 1 -- 172.21.15.111:0/2720925416 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e00106030 con 0x7f1e001054c0 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.363+0000 7f1e052b9640 1 --2- 172.21.15.111:0/2720925416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e001058c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009820 tx=0x7f1dfc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787513623ac4c5c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.363+0000 7f1df7fff640 1 -- 172.21.15.111:0/2720925416 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dfc037070 con 0x7f1e001054c0 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.363+0000 7f1df7fff640 1 -- 172.21.15.111:0/2720925416 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f1dfc032890 con 0x7f1e001054c0 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.363+0000 7f1df7fff640 1 -- 172.21.15.111:0/2720925416 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dfc030550 con 0x7f1e001054c0 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.364+0000 7f1e0751d640 1 -- 172.21.15.111:0/2720925416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 msgr2=0x7f1e001058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.364+0000 7f1e0751d640 1 --2- 172.21.15.111:0/2720925416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e001058c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009820 tx=0x7f1dfc02a270 comp rx=0 tx=0).stop 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.364+0000 7f1e0751d640 1 -- 172.21.15.111:0/2720925416 shutdown_connections 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.364+0000 7f1e0751d640 1 --2- 172.21.15.111:0/2720925416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e001058c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.364+0000 7f1e0751d640 1 -- 172.21.15.111:0/2720925416 >> 172.21.15.111:0/2720925416 conn(0x7f1e00100df0 msgr2=0x7f1e00103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:11.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.364+0000 7f1e0751d640 1 -- 172.21.15.111:0/2720925416 shutdown_connections 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.364+0000 7f1e0751d640 1 -- 172.21.15.111:0/2720925416 wait complete. 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1e0751d640 1 Processor -- start 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1e0751d640 1 -- start start 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1e0751d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e000797d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1e0751d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00109940 con 0x7f1e001054c0 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1e052b9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e000797d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1e052b9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e000797d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35792/0 (socket says 172.21.15.111:35792) 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1e052b9640 1 -- 172.21.15.111:0/673614668 learned_addr learned my addr 172.21.15.111:0/673614668 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1e052b9640 1 -- 172.21.15.111:0/673614668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e00079d30 con 0x7f1e001054c0 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1e052b9640 1 --2- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e000797d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0097f0 tx=0x7f1dfc030ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1df67fc640 1 -- 172.21.15.111:0/673614668 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dfc037040 con 0x7f1e001054c0 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.365+0000 7f1df67fc640 1 -- 172.21.15.111:0/673614668 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f1dfc040070 con 0x7f1e001054c0 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.366+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e0019d360 con 0x7f1e001054c0 2024-09-16T08:54:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.366+0000 7f1df67fc640 1 -- 172.21.15.111:0/673614668 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dfc03bd00 con 0x7f1e001054c0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.366+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e0019d830 con 0x7f1e001054c0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.366+0000 7f1df67fc640 1 -- 172.21.15.111:0/673614668 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) ==== 51778+0+0 (secure 0 0 0) 0x7f1dfc03b430 con 0x7f1e001054c0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.366+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e00109ac0 con 0x7f1e001054c0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.367+0000 7f1df67fc640 1 --2- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f1dec03f6a0 0x7f1dec041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.367+0000 7f1df67fc640 1 -- 172.21.15.111:0/673614668 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1dfc0742d0 con 0x7f1e001054c0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.367+0000 7f1e04ab8640 1 --2- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f1dec03f6a0 0x7f1dec041b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.367+0000 7f1e04ab8640 1 --2- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f1dec03f6a0 0x7f1dec041b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1df00098c0 tx=0x7f1df0006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.372+0000 7f1df67fc640 1 -- 172.21.15.111:0/673614668 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f1dfc0313a0 con 0x7f1e001054c0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:09.564+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}) -- 0x7f1e001021e0 con 0x7f1dec03f6a0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.249+0000 7f1df67fc640 1 -- 172.21.15.111:0/673614668 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+5166 (secure 0 0 0) 0x7f1e001021e0 con 0x7f1dec03f6a0 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.251+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f1dec03f6a0 msgr2=0x7f1dec041b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.251+0000 7f1e0751d640 1 --2- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f1dec03f6a0 0x7f1dec041b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1df00098c0 tx=0x7f1df0006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.251+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 msgr2=0x7f1e000797d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:11.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.251+0000 7f1e0751d640 1 --2- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e000797d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0097f0 tx=0x7f1dfc030ad0 comp rx=0 tx=0).stop 2024-09-16T08:54:11.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.251+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 shutdown_connections 2024-09-16T08:54:11.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.251+0000 7f1e0751d640 1 --2- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f1dec03f6a0 0x7f1dec041b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:11.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.251+0000 7f1e0751d640 1 --2- 172.21.15.111:0/673614668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e001054c0 0x7f1e000797d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:11.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.251+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 >> 172.21.15.111:0/673614668 conn(0x7f1e00100df0 msgr2=0x7f1e00101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:11.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.252+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 shutdown_connections 2024-09-16T08:54:11.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:10.252+0000 7f1e0751d640 1 -- 172.21.15.111:0/673614668 wait complete. 2024-09-16T08:54:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-16T08:54:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.067+0000 7f1552498640 1 Processor -- start 2024-09-16T08:54:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.067+0000 7f1552498640 1 -- start start 2024-09-16T08:54:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.067+0000 7f1552498640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c10b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.067+0000 7f1552498640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f154c07a700 con 0x7f154c07a300 2024-09-16T08:54:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.067+0000 7f154bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c10b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.067+0000 7f154bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c10b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35808/0 (socket says 172.21.15.111:35808) 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.067+0000 7f154bfff640 1 -- 172.21.15.111:0/1917989225 learned_addr learned my addr 172.21.15.111:0/1917989225 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.068+0000 7f154bfff640 1 -- 172.21.15.111:0/1917989225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154c10bb50 con 0x7f154c07a300 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.068+0000 7f154bfff640 1 --2- 172.21.15.111:0/1917989225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c10b610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f153c009820 tx=0x7f153c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8080ec9b1e922e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.068+0000 7f154affd640 1 -- 172.21.15.111:0/1917989225 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f153c037070 con 0x7f154c07a300 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.068+0000 7f154affd640 1 -- 172.21.15.111:0/1917989225 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f153c032890 con 0x7f154c07a300 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.068+0000 7f154affd640 1 -- 172.21.15.111:0/1917989225 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f153c030550 con 0x7f154c07a300 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.069+0000 7f1552498640 1 -- 172.21.15.111:0/1917989225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 msgr2=0x7f154c10b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.069+0000 7f1552498640 1 --2- 172.21.15.111:0/1917989225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c10b610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f153c009820 tx=0x7f153c02a270 comp rx=0 tx=0).stop 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.069+0000 7f1552498640 1 -- 172.21.15.111:0/1917989225 shutdown_connections 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.069+0000 7f1552498640 1 --2- 172.21.15.111:0/1917989225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c10b610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.069+0000 7f1552498640 1 -- 172.21.15.111:0/1917989225 >> 172.21.15.111:0/1917989225 conn(0x7f154c100e00 msgr2=0x7f154c103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.069+0000 7f1552498640 1 -- 172.21.15.111:0/1917989225 shutdown_connections 2024-09-16T08:54:12.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.069+0000 7f1552498640 1 -- 172.21.15.111:0/1917989225 wait complete. 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.070+0000 7f1552498640 1 Processor -- start 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.070+0000 7f1552498640 1 -- start start 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.070+0000 7f1552498640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.070+0000 7f1552498640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f154c10cba0 con 0x7f154c07a300 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.070+0000 7f154bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.070+0000 7f154bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c19c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35812/0 (socket says 172.21.15.111:35812) 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.070+0000 7f154bfff640 1 -- 172.21.15.111:0/1742718281 learned_addr learned my addr 172.21.15.111:0/1742718281 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.070+0000 7f154bfff640 1 -- 172.21.15.111:0/1742718281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154c19cd30 con 0x7f154c07a300 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.070+0000 7f154bfff640 1 --2- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c19c7d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f153c0097f0 tx=0x7f153c030ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.071+0000 7f15497fa640 1 -- 172.21.15.111:0/1742718281 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f153c037040 con 0x7f154c07a300 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.071+0000 7f15497fa640 1 -- 172.21.15.111:0/1742718281 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f153c040070 con 0x7f154c07a300 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.071+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154c19cfc0 con 0x7f154c07a300 2024-09-16T08:54:12.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.071+0000 7f15497fa640 1 -- 172.21.15.111:0/1742718281 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f153c03bd40 con 0x7f154c07a300 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.071+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f154c19d980 con 0x7f154c07a300 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.072+0000 7f15497fa640 1 -- 172.21.15.111:0/1742718281 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) ==== 51778+0+0 (secure 0 0 0) 0x7f153c03b430 con 0x7f154c07a300 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.072+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f154c10b610 con 0x7f154c07a300 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.072+0000 7f15497fa640 1 --2- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f153403f700 0x7f1534041bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.072+0000 7f15497fa640 1 -- 172.21.15.111:0/1742718281 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f153c0742d0 con 0x7f154c07a300 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.072+0000 7f154b7fe640 1 --2- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f153403f700 0x7f1534041bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.072+0000 7f154b7fe640 1 --2- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f153403f700 0x7f1534041bc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15400098c0 tx=0x7f1540006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.076+0000 7f15497fa640 1 -- 172.21.15.111:0/1742718281 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f153c0313a0 con 0x7f154c07a300 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.269+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}) -- 0x7f154c102050 con 0x7f153403f700 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.273+0000 7f15497fa640 1 -- 172.21.15.111:0/1742718281 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f154c102050 con 0x7f153403f700 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f153403f700 msgr2=0x7f1534041bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 --2- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f153403f700 0x7f1534041bc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15400098c0 tx=0x7f1540006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:12.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 msgr2=0x7f154c19c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:12.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 --2- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c19c7d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f153c0097f0 tx=0x7f153c030ad0 comp rx=0 tx=0).stop 2024-09-16T08:54:12.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 shutdown_connections 2024-09-16T08:54:12.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 --2- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f153403f700 0x7f1534041bc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:12.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 --2- 172.21.15.111:0/1742718281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f154c07a300 0x7f154c19c7d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:12.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 >> 172.21.15.111:0/1742718281 conn(0x7f154c100e00 msgr2=0x7f154c101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:12.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 shutdown_connections 2024-09-16T08:54:12.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:12.275+0000 7f1552498640 1 -- 172.21.15.111:0/1742718281 wait complete. 2024-09-16T08:54:13.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:13 smithi111 ceph-mon[29803]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:13.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:13 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:14.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-16T08:54:14.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.286+0000 7fb3ee9f1640 1 Processor -- start 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.286+0000 7fb3ee9f1640 1 -- start start 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.286+0000 7fb3ee9f1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e8106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.286+0000 7fb3ee9f1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8107000 con 0x7fb3e8106610 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.286+0000 7fb3e7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e8106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.286+0000 7fb3e7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e8106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35816/0 (socket says 172.21.15.111:35816) 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.286+0000 7fb3e7fff640 1 -- 172.21.15.111:0/3730538832 learned_addr learned my addr 172.21.15.111:0/3730538832 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.286+0000 7fb3e7fff640 1 -- 172.21.15.111:0/3730538832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e8107180 con 0x7fb3e8106610 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.287+0000 7fb3e7fff640 1 --2- 172.21.15.111:0/3730538832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e8106a10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009820 tx=0x7fb3d802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f5deb8280d43b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.287+0000 7fb3e6ffd640 1 -- 172.21.15.111:0/3730538832 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3d8037070 con 0x7fb3e8106610 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.287+0000 7fb3e6ffd640 1 -- 172.21.15.111:0/3730538832 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fb3d8032890 con 0x7fb3e8106610 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.287+0000 7fb3e6ffd640 1 -- 172.21.15.111:0/3730538832 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3d8030550 con 0x7fb3e8106610 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.288+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/3730538832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 msgr2=0x7fb3e8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:14.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.288+0000 7fb3ee9f1640 1 --2- 172.21.15.111:0/3730538832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e8106a10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009820 tx=0x7fb3d802a270 comp rx=0 tx=0).stop 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.288+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/3730538832 shutdown_connections 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.288+0000 7fb3ee9f1640 1 --2- 172.21.15.111:0/3730538832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e8106a10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.288+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/3730538832 >> 172.21.15.111:0/3730538832 conn(0x7fb3e8101f00 msgr2=0x7fb3e8104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.288+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/3730538832 shutdown_connections 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.288+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/3730538832 wait complete. 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.289+0000 7fb3ee9f1640 1 Processor -- start 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.289+0000 7fb3ee9f1640 1 -- start start 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.289+0000 7fb3ee9f1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.289+0000 7fb3ee9f1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e810aa90 con 0x7fb3e8106610 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.289+0000 7fb3e7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.289+0000 7fb3e7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e819a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35820/0 (socket says 172.21.15.111:35820) 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.289+0000 7fb3e7fff640 1 -- 172.21.15.111:0/2345098635 learned_addr learned my addr 172.21.15.111:0/2345098635 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.289+0000 7fb3e7fff640 1 -- 172.21.15.111:0/2345098635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e819ab30 con 0x7fb3e8106610 2024-09-16T08:54:14.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.289+0000 7fb3e7fff640 1 --2- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e819a5d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80097f0 tx=0x7fb3d8030ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.290+0000 7fb3e57fa640 1 -- 172.21.15.111:0/2345098635 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3d8037040 con 0x7fb3e8106610 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.290+0000 7fb3e57fa640 1 -- 172.21.15.111:0/2345098635 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fb3d8040070 con 0x7fb3e8106610 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.290+0000 7fb3e57fa640 1 -- 172.21.15.111:0/2345098635 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3d803bd40 con 0x7fb3e8106610 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.290+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e819adc0 con 0x7fb3e8106610 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.290+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e819b780 con 0x7fb3e8106610 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.290+0000 7fb3e57fa640 1 -- 172.21.15.111:0/2345098635 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) ==== 51778+0+0 (secure 0 0 0) 0x7fb3d803b430 con 0x7fb3e8106610 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.291+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3e810ac10 con 0x7fb3e8106610 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.291+0000 7fb3e57fa640 1 --2- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb3d003f6a0 0x7fb3d0041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.291+0000 7fb3e57fa640 1 -- 172.21.15.111:0/2345098635 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb3d8074300 con 0x7fb3e8106610 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.291+0000 7fb3e77fe640 1 --2- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb3d003f6a0 0x7fb3d0041b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.291+0000 7fb3e77fe640 1 --2- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb3d003f6a0 0x7fb3d0041b60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0098c0 tx=0x7fb3dc006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.295+0000 7fb3e57fa640 1 -- 172.21.15.111:0/2345098635 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fb3d802e200 con 0x7fb3e8106610 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.491+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}) -- 0x7fb3e8106a10 con 0x7fb3d003f6a0 2024-09-16T08:54:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.494+0000 7fb3e57fa640 1 -- 172.21.15.111:0/2345098635 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fb3e8106a10 con 0x7fb3d003f6a0 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.496+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb3d003f6a0 msgr2=0x7fb3d0041b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.496+0000 7fb3ee9f1640 1 --2- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb3d003f6a0 0x7fb3d0041b60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0098c0 tx=0x7fb3dc006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.496+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 msgr2=0x7fb3e819a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.496+0000 7fb3ee9f1640 1 --2- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e819a5d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80097f0 tx=0x7fb3d8030ad0 comp rx=0 tx=0).stop 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.497+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 shutdown_connections 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.497+0000 7fb3ee9f1640 1 --2- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb3d003f6a0 0x7fb3d0041b60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.497+0000 7fb3ee9f1640 1 --2- 172.21.15.111:0/2345098635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8106610 0x7fb3e819a5d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.497+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 >> 172.21.15.111:0/2345098635 conn(0x7fb3e8101f00 msgr2=0x7fb3e8102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.497+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 shutdown_connections 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:13.497+0000 7fb3ee9f1640 1 -- 172.21.15.111:0/2345098635 wait complete. 2024-09-16T08:54:14.120 INFO:teuthology.orchestra.run.smithi111.stdout:Creating initial admin user... 2024-09-16T08:54:14.752 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:14 smithi111 ceph-mon[29803]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:14.752 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:14 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:15.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$n4K54E.4kz69h4txNnZYJ.BdjsZ6qxqn8O.Fmkw2v.PxwyLSe06T6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726476854, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-16T08:54:15.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.552+0000 7fb2d9842640 1 Processor -- start 2024-09-16T08:54:15.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.552+0000 7fb2d9842640 1 -- start start 2024-09-16T08:54:15.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.552+0000 7fb2d9842640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:15.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.552+0000 7fb2d9842640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d4109710 con 0x7fb2d407a450 2024-09-16T08:54:15.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.552+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:15.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.552+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35832/0 (socket says 172.21.15.111:35832) 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.552+0000 7fb2d2ffd640 1 -- 172.21.15.111:0/2801927876 learned_addr learned my addr 172.21.15.111:0/2801927876 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.552+0000 7fb2d2ffd640 1 -- 172.21.15.111:0/2801927876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d4109890 con 0x7fb2d407a450 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.553+0000 7fb2d2ffd640 1 --2- 172.21.15.111:0/2801927876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d41091d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009820 tx=0x7fb2c402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56cc0e7d829e679 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.553+0000 7fb2d1ffb640 1 -- 172.21.15.111:0/2801927876 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2c4037070 con 0x7fb2d407a450 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.553+0000 7fb2d1ffb640 1 -- 172.21.15.111:0/2801927876 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fb2c4032890 con 0x7fb2d407a450 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.553+0000 7fb2d1ffb640 1 -- 172.21.15.111:0/2801927876 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2c40305d0 con 0x7fb2d407a450 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.554+0000 7fb2d9842640 1 -- 172.21.15.111:0/2801927876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 msgr2=0x7fb2d41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.554+0000 7fb2d9842640 1 --2- 172.21.15.111:0/2801927876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d41091d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009820 tx=0x7fb2c402a300 comp rx=0 tx=0).stop 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.554+0000 7fb2d9842640 1 -- 172.21.15.111:0/2801927876 shutdown_connections 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.554+0000 7fb2d9842640 1 --2- 172.21.15.111:0/2801927876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d41091d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.554+0000 7fb2d9842640 1 -- 172.21.15.111:0/2801927876 >> 172.21.15.111:0/2801927876 conn(0x7fb2d4100e00 msgr2=0x7fb2d4103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.554+0000 7fb2d9842640 1 -- 172.21.15.111:0/2801927876 shutdown_connections 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.554+0000 7fb2d9842640 1 -- 172.21.15.111:0/2801927876 wait complete. 2024-09-16T08:54:15.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.554+0000 7fb2d9842640 1 Processor -- start 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d9842640 1 -- start start 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d9842640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d9842640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d410a940 con 0x7fb2d407a450 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d2ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d419a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35840/0 (socket says 172.21.15.111:35840) 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d2ffd640 1 -- 172.21.15.111:0/955546996 learned_addr learned my addr 172.21.15.111:0/955546996 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d2ffd640 1 -- 172.21.15.111:0/955546996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d419ab70 con 0x7fb2d407a450 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d2ffd640 1 --2- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d419a610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4032b60 tx=0x7fb2c4030980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d8840640 1 -- 172.21.15.111:0/955546996 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2c4037040 con 0x7fb2d407a450 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.555+0000 7fb2d8840640 1 -- 172.21.15.111:0/955546996 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fb2c4040070 con 0x7fb2d407a450 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.556+0000 7fb2d8840640 1 -- 172.21.15.111:0/955546996 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2c403bc90 con 0x7fb2d407a450 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.556+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d419ae00 con 0x7fb2d407a450 2024-09-16T08:54:15.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.556+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d419b7c0 con 0x7fb2d407a450 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.556+0000 7fb2d8840640 1 -- 172.21.15.111:0/955546996 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) ==== 51778+0+0 (secure 0 0 0) 0x7fb2c403b430 con 0x7fb2d407a450 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.557+0000 7fb2d8840640 1 --2- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb2bc03f6a0 0x7fb2bc041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.557+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2d4107240 con 0x7fb2d407a450 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.557+0000 7fb2d8840640 1 -- 172.21.15.111:0/955546996 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb2c4074150 con 0x7fb2d407a450 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.557+0000 7fb2d27fc640 1 --2- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb2bc03f6a0 0x7fb2bc041b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.557+0000 7fb2d27fc640 1 --2- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb2bc03f6a0 0x7fb2bc041b60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80098c0 tx=0x7fb2c8006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.561+0000 7fb2d8840640 1 -- 172.21.15.111:0/955546996 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fb2c40313a0 con 0x7fb2d407a450 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.751+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fb2d41020f0 con 0x7fb2bc03f6a0 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.974+0000 7fb2d8840640 1 -- 172.21.15.111:0/955546996 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fb2d41020f0 con 0x7fb2bc03f6a0 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb2bc03f6a0 msgr2=0x7fb2bc041b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 --2- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb2bc03f6a0 0x7fb2bc041b60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80098c0 tx=0x7fb2c8006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 msgr2=0x7fb2d419a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:15.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 --2- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d419a610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4032b60 tx=0x7fb2c4030980 comp rx=0 tx=0).stop 2024-09-16T08:54:15.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 shutdown_connections 2024-09-16T08:54:15.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 --2- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb2bc03f6a0 0x7fb2bc041b60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:15.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 --2- 172.21.15.111:0/955546996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d407a450 0x7fb2d419a610 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:15.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 >> 172.21.15.111:0/955546996 conn(0x7fb2d4100e00 msgr2=0x7fb2d41019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:15.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 shutdown_connections 2024-09-16T08:54:15.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:14.976+0000 7fb2d9842640 1 -- 172.21.15.111:0/955546996 wait complete. 2024-09-16T08:54:15.489 INFO:teuthology.orchestra.run.smithi111.stdout:Fetching dashboard port number... 2024-09-16T08:54:15.574 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:15 smithi111 ceph-mon[29803]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:15.575 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:15 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1493710779' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-16T08:54:16.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 8443 2024-09-16T08:54:16.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.917+0000 7f54434cf640 1 Processor -- start 2024-09-16T08:54:16.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.918+0000 7f54434cf640 1 -- start start 2024-09-16T08:54:16.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.918+0000 7f54434cf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.918+0000 7f54434cf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f543c1097d0 con 0x7f543c07a390 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.918+0000 7f544126b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.918+0000 7f544126b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35852/0 (socket says 172.21.15.111:35852) 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.918+0000 7f544126b640 1 -- 172.21.15.111:0/1795815809 learned_addr learned my addr 172.21.15.111:0/1795815809 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.918+0000 7f544126b640 1 -- 172.21.15.111:0/1795815809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543c109950 con 0x7f543c07a390 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.919+0000 7f544126b640 1 --2- 172.21.15.111:0/1795815809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c109290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5438009820 tx=0x7f543802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0643bd447b53b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.919+0000 7f5433fff640 1 -- 172.21.15.111:0/1795815809 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5438037070 con 0x7f543c07a390 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.919+0000 7f5433fff640 1 -- 172.21.15.111:0/1795815809 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f5438032890 con 0x7f543c07a390 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.919+0000 7f5433fff640 1 -- 172.21.15.111:0/1795815809 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5438030550 con 0x7f543c07a390 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 -- 172.21.15.111:0/1795815809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 msgr2=0x7f543c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 --2- 172.21.15.111:0/1795815809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c109290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5438009820 tx=0x7f543802a270 comp rx=0 tx=0).stop 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 -- 172.21.15.111:0/1795815809 shutdown_connections 2024-09-16T08:54:16.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 --2- 172.21.15.111:0/1795815809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c109290 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 -- 172.21.15.111:0/1795815809 >> 172.21.15.111:0/1795815809 conn(0x7f543c100e00 msgr2=0x7f543c103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 -- 172.21.15.111:0/1795815809 shutdown_connections 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 -- 172.21.15.111:0/1795815809 wait complete. 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 Processor -- start 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 -- start start 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c1068e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.920+0000 7f54434cf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f543c10a8e0 con 0x7f543c07a390 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f544126b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c1068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f544126b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c1068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35854/0 (socket says 172.21.15.111:35854) 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f544126b640 1 -- 172.21.15.111:0/1493710779 learned_addr learned my addr 172.21.15.111:0/1493710779 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f544126b640 1 -- 172.21.15.111:0/1493710779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543c106e40 con 0x7f543c07a390 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f544126b640 1 --2- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c1068e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5438032860 tx=0x7f5438030a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f54327fc640 1 -- 172.21.15.111:0/1493710779 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5438037040 con 0x7f543c07a390 2024-09-16T08:54:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f54327fc640 1 -- 172.21.15.111:0/1493710779 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f5438040070 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543c105230 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f54327fc640 1 -- 172.21.15.111:0/1493710779 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f543803bd60 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.921+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f543c105810 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.922+0000 7f54327fc640 1 -- 172.21.15.111:0/1493710779 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) ==== 51778+0+0 (secure 0 0 0) 0x7f5438045460 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.922+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543c107070 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.922+0000 7f54327fc640 1 --2- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f542803f700 0x7f5428041bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.922+0000 7f54327fc640 1 -- 172.21.15.111:0/1493710779 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5438074420 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.922+0000 7f5440a6a640 1 --2- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f542803f700 0x7f5428041bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.923+0000 7f5440a6a640 1 --2- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f542803f700 0x7f5428041bc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f542c0098c0 tx=0x7f542c006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:15.927+0000 7f54327fc640 1 -- 172.21.15.111:0/1493710779 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f5438030dc0 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.103+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f543c105cf0 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.103+0000 7f54327fc640 1 -- 172.21.15.111:0/1493710779 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f5438042d20 con 0x7f543c07a390 2024-09-16T08:54:16.701 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f542803f700 msgr2=0x7f5428041bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:16.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 --2- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f542803f700 0x7f5428041bc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f542c0098c0 tx=0x7f542c006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:16.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 msgr2=0x7f543c1068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:16.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 --2- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c1068e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5438032860 tx=0x7f5438030a80 comp rx=0 tx=0).stop 2024-09-16T08:54:16.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 shutdown_connections 2024-09-16T08:54:16.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 --2- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f542803f700 0x7f5428041bc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:16.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 --2- 172.21.15.111:0/1493710779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f543c07a390 0x7f543c1068e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:16.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 >> 172.21.15.111:0/1493710779 conn(0x7f543c100e00 msgr2=0x7f543c101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:16.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 shutdown_connections 2024-09-16T08:54:16.702 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:16.106+0000 7f54434cf640 1 -- 172.21.15.111:0/1493710779 wait complete. 2024-09-16T08:54:16.703 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T08:54:16.703 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2024-09-16T08:54:16.710 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T08:54:16.710 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-09-16T08:54:16.710 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2024-09-16T08:54:16.710 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-16T08:54:16.711 INFO:teuthology.orchestra.run.smithi111.stdout:Ceph Dashboard is now available at: 2024-09-16T08:54:16.711 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:16.712 INFO:teuthology.orchestra.run.smithi111.stdout: URL: https://smithi111.front.sepia.ceph.com:8443/ 2024-09-16T08:54:16.712 INFO:teuthology.orchestra.run.smithi111.stdout: User: admin 2024-09-16T08:54:16.712 INFO:teuthology.orchestra.run.smithi111.stdout: Password: cprjueyekx 2024-09-16T08:54:16.712 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:16.712 INFO:teuthology.orchestra.run.smithi111.stdout:Saving cluster configuration to /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config directory 2024-09-16T08:54:17.977 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.128+0000 7fbf151ae640 1 Processor -- start 2024-09-16T08:54:17.977 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.128+0000 7fbf151ae640 1 -- start start 2024-09-16T08:54:17.977 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.128+0000 7fbf151ae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf10106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.128+0000 7fbf151ae640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbf10107000 con 0x7fbf10106610 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.128+0000 7fbf0ed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf10106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.128+0000 7fbf0ed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf10106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35864/0 (socket says 172.21.15.111:35864) 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.128+0000 7fbf0ed9d640 1 -- 172.21.15.111:0/3128573640 learned_addr learned my addr 172.21.15.111:0/3128573640 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.129+0000 7fbf0ed9d640 1 -- 172.21.15.111:0/3128573640 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf10107180 con 0x7fbf10106610 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.129+0000 7fbf0ed9d640 1 --2- 172.21.15.111:0/3128573640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf10106a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf00009e80 tx=0x7fbf0002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ad6e67dba175d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf0dd9b640 1 -- 172.21.15.111:0/3128573640 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf00036070 con 0x7fbf10106610 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf0dd9b640 1 -- 172.21.15.111:0/3128573640 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fbf00031850 con 0x7fbf10106610 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf0dd9b640 1 -- 172.21.15.111:0/3128573640 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf000076d0 con 0x7fbf10106610 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf151ae640 1 -- 172.21.15.111:0/3128573640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 msgr2=0x7fbf10106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf151ae640 1 --2- 172.21.15.111:0/3128573640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf10106a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf00009e80 tx=0x7fbf0002a7a0 comp rx=0 tx=0).stop 2024-09-16T08:54:17.978 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf151ae640 1 -- 172.21.15.111:0/3128573640 shutdown_connections 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf151ae640 1 --2- 172.21.15.111:0/3128573640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf10106a10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf151ae640 1 -- 172.21.15.111:0/3128573640 >> 172.21.15.111:0/3128573640 conn(0x7fbf10101f00 msgr2=0x7fbf10104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf151ae640 1 -- 172.21.15.111:0/3128573640 shutdown_connections 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.130+0000 7fbf151ae640 1 -- 172.21.15.111:0/3128573640 wait complete. 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.131+0000 7fbf151ae640 1 Processor -- start 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.131+0000 7fbf151ae640 1 -- start start 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.131+0000 7fbf151ae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf1007c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.131+0000 7fbf151ae640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1010aa90 con 0x7fbf10106610 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.131+0000 7fbf0ed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf1007c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.131+0000 7fbf0ed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf1007c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35874/0 (socket says 172.21.15.111:35874) 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.131+0000 7fbf0ed9d640 1 -- 172.21.15.111:0/4238811974 learned_addr learned my addr 172.21.15.111:0/4238811974 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.132+0000 7fbf0ed9d640 1 -- 172.21.15.111:0/4238811974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1007c6e0 con 0x7fbf10106610 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.132+0000 7fbf0ed9d640 1 --2- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf1007c180 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbf00000f80 tx=0x7fbf00031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:17.979 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.132+0000 7fbf07fff640 1 -- 172.21.15.111:0/4238811974 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf00041020 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.132+0000 7fbf07fff640 1 -- 172.21.15.111:0/4238811974 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fbf00007c00 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.132+0000 7fbf07fff640 1 -- 172.21.15.111:0/4238811974 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf00036040 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.132+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1007aad0 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.132+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf1007b0b0 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.133+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf1010ac10 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.133+0000 7fbf07fff640 1 -- 172.21.15.111:0/4238811974 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) ==== 51778+0+0 (secure 0 0 0) 0x7fbf00044020 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.134+0000 7fbf07fff640 1 --2- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbefc03f6a0 0x7fbefc041b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.134+0000 7fbf07fff640 1 -- 172.21.15.111:0/4238811974 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbf000752e0 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.134+0000 7fbf0e59c640 1 --2- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbefc03f6a0 0x7fbefc041b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.134+0000 7fbf0e59c640 1 --2- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbefc03f6a0 0x7fbefc041b60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbef80098c0 tx=0x7fbef8006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.138+0000 7fbf07fff640 1 -- 172.21.15.111:0/4238811974 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fbf0004b020 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.415+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fbf1007b280 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.419+0000 7fbf07fff640 1 -- 172.21.15.111:0/4238811974 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fbf0004c260 con 0x7fbf10106610 2024-09-16T08:54:17.980 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.421+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbefc03f6a0 msgr2=0x7fbefc041b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.421+0000 7fbf151ae640 1 --2- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbefc03f6a0 0x7fbefc041b60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbef80098c0 tx=0x7fbef8006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.421+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 msgr2=0x7fbf1007c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.421+0000 7fbf151ae640 1 --2- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf1007c180 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbf00000f80 tx=0x7fbf00031700 comp rx=0 tx=0).stop 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.421+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 shutdown_connections 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.421+0000 7fbf151ae640 1 --2- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbefc03f6a0 0x7fbefc041b60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.421+0000 7fbf151ae640 1 --2- 172.21.15.111:0/4238811974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf10106610 0x7fbf1007c180 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.421+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 >> 172.21.15.111:0/4238811974 conn(0x7fbf10101f00 msgr2=0x7fbf10102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.421+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 shutdown_connections 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-09-16T08:54:17.422+0000 7fbf151ae640 1 -- 172.21.15.111:0/4238811974 wait complete. 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-16T08:54:17.981 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout:Or, if you are only running a single cluster on this host: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout: ceph telemetry on 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout:For more information see: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:17.982 INFO:teuthology.orchestra.run.smithi111.stdout:Bootstrap complete. 2024-09-16T08:54:18.020 INFO:tasks.cephadm:Fetching config... 2024-09-16T08:54:18.020 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:54:18.020 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-16T08:54:18.036 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-16T08:54:18.036 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:54:18.036 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-16T08:54:18.092 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-16T08:54:18.092 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:54:18.092 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-16T08:54:18.157 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-16T08:54:18.157 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:54:18.157 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-16T08:54:18.213 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-16T08:54:18.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5VnMGKtwHDkwCxDpc3F9Xr0xhgkhmDDlYaz+iGkUhJLHNkOxsU7sgKobQZ3Rx/CkjMe6mE6FkGtTx1scjSC25nEkRvjXUKd4eIBW5XucVEPOYezXnglh/U65v1gdLHJUZJcqW3KiQehqIHrqfQFtSKT41i8L/5Y+UL9DklOPaKpQUiyvRZW5O7RGufGOO6xcUpAUNKYwCLlkiiEV4Ru8Fs21gpFLYuthcpN/n0B18brnzNvF92lsxS6w+rggIEabeDS1Pw9vD0Rovu9qtjC6qbANf0Ju0yGNN68W94kGsZgq0UM+ydJsk+/jZhNMQipoXKL453pDX+H2IUTqc7zdGfX8UJz/VnFZHQt6HbhuFP0kxevVt96zpAa/GK3TvPcPbHAgDLSpORhBP8wr5Rj5YsUPzvPQ3H/KcG5BQKZDSZIQDs15Ph9qHDWEagXEEiRf3vzGxZb5V4dIJex6yFTYvbi5eG1ggAqZCzxsUbnPY9pbNP+HWb7HjErD5nHKjyLE= ceph-f774c998-7408-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T08:54:18.290 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5VnMGKtwHDkwCxDpc3F9Xr0xhgkhmDDlYaz+iGkUhJLHNkOxsU7sgKobQZ3Rx/CkjMe6mE6FkGtTx1scjSC25nEkRvjXUKd4eIBW5XucVEPOYezXnglh/U65v1gdLHJUZJcqW3KiQehqIHrqfQFtSKT41i8L/5Y+UL9DklOPaKpQUiyvRZW5O7RGufGOO6xcUpAUNKYwCLlkiiEV4Ru8Fs21gpFLYuthcpN/n0B18brnzNvF92lsxS6w+rggIEabeDS1Pw9vD0Rovu9qtjC6qbANf0Ju0yGNN68W94kGsZgq0UM+ydJsk+/jZhNMQipoXKL453pDX+H2IUTqc7zdGfX8UJz/VnFZHQt6HbhuFP0kxevVt96zpAa/GK3TvPcPbHAgDLSpORhBP8wr5Rj5YsUPzvPQ3H/KcG5BQKZDSZIQDs15Ph9qHDWEagXEEiRf3vzGxZb5V4dIJex6yFTYvbi5eG1ggAqZCzxsUbnPY9pbNP+HWb7HjErD5nHKjyLE= ceph-f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:18.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5VnMGKtwHDkwCxDpc3F9Xr0xhgkhmDDlYaz+iGkUhJLHNkOxsU7sgKobQZ3Rx/CkjMe6mE6FkGtTx1scjSC25nEkRvjXUKd4eIBW5XucVEPOYezXnglh/U65v1gdLHJUZJcqW3KiQehqIHrqfQFtSKT41i8L/5Y+UL9DklOPaKpQUiyvRZW5O7RGufGOO6xcUpAUNKYwCLlkiiEV4Ru8Fs21gpFLYuthcpN/n0B18brnzNvF92lsxS6w+rggIEabeDS1Pw9vD0Rovu9qtjC6qbANf0Ju0yGNN68W94kGsZgq0UM+ydJsk+/jZhNMQipoXKL453pDX+H2IUTqc7zdGfX8UJz/VnFZHQt6HbhuFP0kxevVt96zpAa/GK3TvPcPbHAgDLSpORhBP8wr5Rj5YsUPzvPQ3H/KcG5BQKZDSZIQDs15Ph9qHDWEagXEEiRf3vzGxZb5V4dIJex6yFTYvbi5eG1ggAqZCzxsUbnPY9pbNP+HWb7HjErD5nHKjyLE= ceph-f774c998-7408-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T08:54:18.417 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5VnMGKtwHDkwCxDpc3F9Xr0xhgkhmDDlYaz+iGkUhJLHNkOxsU7sgKobQZ3Rx/CkjMe6mE6FkGtTx1scjSC25nEkRvjXUKd4eIBW5XucVEPOYezXnglh/U65v1gdLHJUZJcqW3KiQehqIHrqfQFtSKT41i8L/5Y+UL9DklOPaKpQUiyvRZW5O7RGufGOO6xcUpAUNKYwCLlkiiEV4Ru8Fs21gpFLYuthcpN/n0B18brnzNvF92lsxS6w+rggIEabeDS1Pw9vD0Rovu9qtjC6qbANf0Ju0yGNN68W94kGsZgq0UM+ydJsk+/jZhNMQipoXKL453pDX+H2IUTqc7zdGfX8UJz/VnFZHQt6HbhuFP0kxevVt96zpAa/GK3TvPcPbHAgDLSpORhBP8wr5Rj5YsUPzvPQ3H/KcG5BQKZDSZIQDs15Ph9qHDWEagXEEiRf3vzGxZb5V4dIJex6yFTYvbi5eG1ggAqZCzxsUbnPY9pbNP+HWb7HjErD5nHKjyLE= ceph-f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:18.430 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-16T08:54:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4238811974' entity='client.admin' 2024-09-16T08:54:18.678 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:54:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.171+0000 7f42dd48c640 1 -- 172.21.15.111:0/1982492036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 msgr2=0x7f42d81016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.171+0000 7f42dd48c640 1 --2- 172.21.15.111:0/1982492036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 0x7f42d81016f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42c8009880 tx=0x7f42c802a5d0 comp rx=0 tx=0).stop 2024-09-16T08:54:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.171+0000 7f42dd48c640 1 -- 172.21.15.111:0/1982492036 shutdown_connections 2024-09-16T08:54:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.171+0000 7f42dd48c640 1 --2- 172.21.15.111:0/1982492036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 0x7f42d81016f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.171+0000 7f42dd48c640 1 -- 172.21.15.111:0/1982492036 >> 172.21.15.111:0/1982492036 conn(0x7f42d80fcbc0 msgr2=0x7f42d80ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.171+0000 7f42dd48c640 1 -- 172.21.15.111:0/1982492036 shutdown_connections 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.171+0000 7f42dd48c640 1 -- 172.21.15.111:0/1982492036 wait complete. 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.172+0000 7f42dd48c640 1 Processor -- start 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.172+0000 7f42dd48c640 1 -- start start 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.172+0000 7f42dd48c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 0x7f42d806aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.172+0000 7f42dd48c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f42d81067c0 con 0x7f42d81012f0 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.172+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 0x7f42d806aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.172+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 0x7f42d806aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35894/0 (socket says 172.21.15.111:35894) 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.172+0000 7f42d6ffd640 1 -- 172.21.15.111:0/2790029804 learned_addr learned my addr 172.21.15.111:0/2790029804 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.173+0000 7f42d6ffd640 1 -- 172.21.15.111:0/2790029804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d806b400 con 0x7f42d81012f0 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.173+0000 7f42d6ffd640 1 --2- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 0x7f42d806aea0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42c8004dc0 tx=0x7f42c8005090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.173+0000 7f42bffff640 1 -- 172.21.15.111:0/2790029804 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42c803f070 con 0x7f42d81012f0 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.173+0000 7f42bffff640 1 -- 172.21.15.111:0/2790029804 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f42c8003a40 con 0x7f42d81012f0 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.173+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d8199eb0 con 0x7f42d81012f0 2024-09-16T08:54:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.173+0000 7f42bffff640 1 -- 172.21.15.111:0/2790029804 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42c8035030 con 0x7f42d81012f0 2024-09-16T08:54:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.173+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d819a390 con 0x7f42d81012f0 2024-09-16T08:54:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.174+0000 7f42bffff640 1 -- 172.21.15.111:0/2790029804 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) ==== 51778+0+0 (secure 0 0 0) 0x7f42c8042070 con 0x7f42d81012f0 2024-09-16T08:54:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.174+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d8106f40 con 0x7f42d81012f0 2024-09-16T08:54:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.174+0000 7f42bffff640 1 --2- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f42c003f700 0x7f42c0041bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.174+0000 7f42bffff640 1 -- 172.21.15.111:0/2790029804 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f42c8074c30 con 0x7f42d81012f0 2024-09-16T08:54:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.174+0000 7f42d67fc640 1 --2- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f42c003f700 0x7f42c0041bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.174+0000 7f42d67fc640 1 --2- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f42c003f700 0x7f42c0041bc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0098c0 tx=0x7f42cc006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.179+0000 7f42bffff640 1 -- 172.21.15.111:0/2790029804 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f42c8031e20 con 0x7f42d81012f0 2024-09-16T08:54:19.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.356+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f42d802e6a0 con 0x7f42d81012f0 2024-09-16T08:54:19.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.358+0000 7f42bffff640 1 -- 172.21.15.111:0/2790029804 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f42c80055a0 con 0x7f42d81012f0 2024-09-16T08:54:19.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f42c003f700 msgr2=0x7f42c0041bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:19.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 --2- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f42c003f700 0x7f42c0041bc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0098c0 tx=0x7f42cc006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:19.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 msgr2=0x7f42d806aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:19.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 --2- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 0x7f42d806aea0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42c8004dc0 tx=0x7f42c8005090 comp rx=0 tx=0).stop 2024-09-16T08:54:19.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 shutdown_connections 2024-09-16T08:54:19.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 --2- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f42c003f700 0x7f42c0041bc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:19.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 --2- 172.21.15.111:0/2790029804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42d81012f0 0x7f42d806aea0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:19.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 >> 172.21.15.111:0/2790029804 conn(0x7f42d80fcbc0 msgr2=0x7f42d80fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:19.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 shutdown_connections 2024-09-16T08:54:19.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:19.361+0000 7f42dd48c640 1 -- 172.21.15.111:0/2790029804 wait complete. 2024-09-16T08:54:20.598 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-16T08:54:20.598 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-16T08:54:20.622 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2790029804' entity='client.admin' 2024-09-16T08:54:20.622 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:20 smithi111 ceph-mon[29803]: mgrmap e13: y(active, since 12s) 2024-09-16T08:54:20.622 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:20.622 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:20.623 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"} : dispatch 2024-09-16T08:54:20.623 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:20.623 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:20.623 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:20.919 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:54:21.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.387+0000 7f3dc5cec640 1 -- 172.21.15.111:0/3659478819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 msgr2=0x7f3dc0101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:21.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.387+0000 7f3dc5cec640 1 --2- 172.21.15.111:0/3659478819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 0x7f3dc0101710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3db0009880 tx=0x7f3db002a5d0 comp rx=0 tx=0).stop 2024-09-16T08:54:21.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.387+0000 7f3dc5cec640 1 -- 172.21.15.111:0/3659478819 shutdown_connections 2024-09-16T08:54:21.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.387+0000 7f3dc5cec640 1 --2- 172.21.15.111:0/3659478819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 0x7f3dc0101710 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:21.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.387+0000 7f3dc5cec640 1 -- 172.21.15.111:0/3659478819 >> 172.21.15.111:0/3659478819 conn(0x7f3dc00fcbe0 msgr2=0x7f3dc00ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:21.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.387+0000 7f3dc5cec640 1 -- 172.21.15.111:0/3659478819 shutdown_connections 2024-09-16T08:54:21.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.387+0000 7f3dc5cec640 1 -- 172.21.15.111:0/3659478819 wait complete. 2024-09-16T08:54:21.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.387+0000 7f3dc5cec640 1 Processor -- start 2024-09-16T08:54:21.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dc5cec640 1 -- start start 2024-09-16T08:54:21.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dc5cec640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 0x7f3dc0196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:21.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dc5cec640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01067e0 con 0x7f3dc0101310 2024-09-16T08:54:21.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dbf7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 0x7f3dc0196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:21.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dbf7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 0x7f3dc0196080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35700/0 (socket says 172.21.15.111:35700) 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dbf7fe640 1 -- 172.21.15.111:0/14408608 learned_addr learned my addr 172.21.15.111:0/14408608 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dbf7fe640 1 -- 172.21.15.111:0/14408608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc01965e0 con 0x7f3dc0101310 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dbf7fe640 1 --2- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 0x7f3dc0196080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3db0005ab0 tx=0x7f3db0005090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dbd7fa640 1 -- 172.21.15.111:0/14408608 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3db003f070 con 0x7f3dc0101310 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dbd7fa640 1 -- 172.21.15.111:0/14408608 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f3db0003b70 con 0x7f3dc0101310 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc0196870 con 0x7f3dc0101310 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dbd7fa640 1 -- 172.21.15.111:0/14408608 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3db0035030 con 0x7f3dc0101310 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.388+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc0197230 con 0x7f3dc0101310 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.389+0000 7f3dbd7fa640 1 -- 172.21.15.111:0/14408608 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7f3db0042070 con 0x7f3dc0101310 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.389+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc0106f60 con 0x7f3dc0101310 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.390+0000 7f3dbd7fa640 1 --2- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f3dac03f7a0 0x7f3dac041c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.390+0000 7f3dbd7fa640 1 -- 172.21.15.111:0/14408608 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3db0074e40 con 0x7f3dc0101310 2024-09-16T08:54:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.390+0000 7f3db7fff640 1 --2- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f3dac03f7a0 0x7f3dac041c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:21.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.390+0000 7f3db7fff640 1 --2- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f3dac03f7a0 0x7f3dac041c60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3da80098c0 tx=0x7f3da8006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:21.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.394+0000 7f3dbd7fa640 1 -- 172.21.15.111:0/14408608 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f3db0039e20 con 0x7f3dc0101310 2024-09-16T08:54:21.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.577+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f3dc0197f70 con 0x7f3dac03f7a0 2024-09-16T08:54:21.581 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.580+0000 7f3dbd7fa640 1 -- 172.21.15.111:0/14408608 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3dc0197f70 con 0x7f3dac03f7a0 2024-09-16T08:54:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f3dac03f7a0 msgr2=0x7f3dac041c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 --2- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f3dac03f7a0 0x7f3dac041c60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3da80098c0 tx=0x7f3da8006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 msgr2=0x7f3dc0196080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 --2- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 0x7f3dc0196080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3db0005ab0 tx=0x7f3db0005090 comp rx=0 tx=0).stop 2024-09-16T08:54:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 shutdown_connections 2024-09-16T08:54:21.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 --2- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f3dac03f7a0 0x7f3dac041c60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:21.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 --2- 172.21.15.111:0/14408608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc0101310 0x7f3dc0196080 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:21.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 >> 172.21.15.111:0/14408608 conn(0x7f3dc00fcbe0 msgr2=0x7f3dc00fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:21.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 shutdown_connections 2024-09-16T08:54:21.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:21.583+0000 7f3dc5cec640 1 -- 172.21.15.111:0/14408608 wait complete. 2024-09-16T08:54:22.586 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:22.586 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:22.586 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:22.586 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:22.586 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:22.586 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: Updating smithi111:/etc/ceph/ceph.conf 2024-09-16T08:54:22.587 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:22.587 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:54:22.587 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:22.587 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:22.587 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:22.611 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi114 2024-09-16T08:54:22.611 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:54:22.611 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.conf 2024-09-16T08:54:22.630 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:54:22.630 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:54:22.688 INFO:tasks.cephadm:Adding host smithi114 to orchestrator... 2024-09-16T08:54:22.689 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch host add smithi114 2024-09-16T08:54:22.927 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:54:23.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.386+0000 7fe53f072640 1 -- 172.21.15.111:0/1000422611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 msgr2=0x7fe538101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:23.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.386+0000 7fe53f072640 1 --2- 172.21.15.111:0/1000422611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 0x7fe538101dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009880 tx=0x7fe52c02a5d0 comp rx=0 tx=0).stop 2024-09-16T08:54:23.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.386+0000 7fe53f072640 1 -- 172.21.15.111:0/1000422611 shutdown_connections 2024-09-16T08:54:23.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.386+0000 7fe53f072640 1 --2- 172.21.15.111:0/1000422611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 0x7fe538101dd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:23.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.386+0000 7fe53f072640 1 -- 172.21.15.111:0/1000422611 >> 172.21.15.111:0/1000422611 conn(0x7fe538077140 msgr2=0x7fe538077550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.386+0000 7fe53f072640 1 -- 172.21.15.111:0/1000422611 shutdown_connections 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.386+0000 7fe53f072640 1 -- 172.21.15.111:0/1000422611 wait complete. 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.386+0000 7fe53f072640 1 Processor -- start 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe53f072640 1 -- start start 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe53f072640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 0x7fe5381952f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe53f072640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe53806b1d0 con 0x7fe5381019f0 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe53ce0e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 0x7fe5381952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe53ce0e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 0x7fe5381952f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35718/0 (socket says 172.21.15.111:35718) 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe53ce0e640 1 -- 172.21.15.111:0/1677539354 learned_addr learned my addr 172.21.15.111:0/1677539354 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe53ce0e640 1 -- 172.21.15.111:0/1677539354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe538195830 con 0x7fe5381019f0 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe53ce0e640 1 --2- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 0x7fe5381952f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe52c005d90 tx=0x7fe52c003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe535ffb640 1 -- 172.21.15.111:0/1677539354 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe52c03f070 con 0x7fe5381019f0 2024-09-16T08:54:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.387+0000 7fe535ffb640 1 -- 172.21.15.111:0/1677539354 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fe52c03bbf0 con 0x7fe5381019f0 2024-09-16T08:54:23.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.388+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe538195b20 con 0x7fe5381019f0 2024-09-16T08:54:23.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.388+0000 7fe535ffb640 1 -- 172.21.15.111:0/1677539354 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe52c035030 con 0x7fe5381019f0 2024-09-16T08:54:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.388+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe53819abd0 con 0x7fe5381019f0 2024-09-16T08:54:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.389+0000 7fe535ffb640 1 -- 172.21.15.111:0/1677539354 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7fe52c042070 con 0x7fe5381019f0 2024-09-16T08:54:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.389+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5381029b0 con 0x7fe5381019f0 2024-09-16T08:54:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.389+0000 7fe535ffb640 1 --2- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe5280419f0 0x7fe528043eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.389+0000 7fe535ffb640 1 -- 172.21.15.111:0/1677539354 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe52c0749d0 con 0x7fe5381019f0 2024-09-16T08:54:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.389+0000 7fe537fff640 1 --2- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe5280419f0 0x7fe528043eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.389+0000 7fe537fff640 1 --2- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe5280419f0 0x7fe528043eb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe5240098c0 tx=0x7fe524006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:23.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.394+0000 7fe535ffb640 1 -- 172.21.15.111:0/1677539354 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fe52c03a3c0 con 0x7fe5381019f0 2024-09-16T08:54:23.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:23.572+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}) -- 0x7fe538101e50 con 0x7fe5280419f0 2024-09-16T08:54:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:23 smithi111 ceph-mon[29803]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:54:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:24 smithi111 ceph-mon[29803]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:25 smithi111 ceph-mon[29803]: Deploying cephadm binary to smithi114 2024-09-16T08:54:26.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.571+0000 7fe535ffb640 1 -- 172.21.15.111:0/1677539354 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fe538101e50 con 0x7fe5280419f0 2024-09-16T08:54:26.572 INFO:teuthology.orchestra.run.smithi111.stdout:Added host 'smithi114' with addr '172.21.15.114' 2024-09-16T08:54:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe5280419f0 msgr2=0x7fe528043eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 --2- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe5280419f0 0x7fe528043eb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe5240098c0 tx=0x7fe524006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 msgr2=0x7fe5381952f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 --2- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 0x7fe5381952f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe52c005d90 tx=0x7fe52c003d60 comp rx=0 tx=0).stop 2024-09-16T08:54:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 shutdown_connections 2024-09-16T08:54:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 --2- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe5280419f0 0x7fe528043eb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 --2- 172.21.15.111:0/1677539354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5381019f0 0x7fe5381952f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 >> 172.21.15.111:0/1677539354 conn(0x7fe538077140 msgr2=0x7fe538107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 shutdown_connections 2024-09-16T08:54:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:26.574+0000 7fe53f072640 1 -- 172.21.15.111:0/1677539354 wait complete. 2024-09-16T08:54:27.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-16T08:54:27.431 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:54:27.858 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:27.859 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:27 smithi111 ceph-mon[29803]: Added host smithi114 2024-09-16T08:54:27.859 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:27.859 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:27.859 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:27.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.857+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1738015876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 msgr2=0x7f0fb0102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.857+0000 7f0fb84b0640 1 --2- 172.21.15.111:0/1738015876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 0x7f0fb0102b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009c20 tx=0x7f0fa002aa90 comp rx=0 tx=0).stop 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.857+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1738015876 shutdown_connections 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.857+0000 7f0fb84b0640 1 --2- 172.21.15.111:0/1738015876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 0x7f0fb0102b90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.857+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1738015876 >> 172.21.15.111:0/1738015876 conn(0x7f0fb00fe220 msgr2=0x7f0fb0100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.858+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1738015876 shutdown_connections 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.858+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1738015876 wait complete. 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.858+0000 7f0fb84b0640 1 Processor -- start 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.858+0000 7f0fb84b0640 1 -- start start 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.858+0000 7f0fb84b0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 0x7f0fb0197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.858+0000 7f0fb84b0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb010bed0 con 0x7f0fb01027b0 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.858+0000 7f0fb624c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 0x7f0fb0197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:27.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.859+0000 7f0fb624c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 0x7f0fb0197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35748/0 (socket says 172.21.15.111:35748) 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.859+0000 7f0fb624c640 1 -- 172.21.15.111:0/1451324566 learned_addr learned my addr 172.21.15.111:0/1451324566 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.859+0000 7f0fb624c640 1 -- 172.21.15.111:0/1451324566 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb0197a70 con 0x7f0fb01027b0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.859+0000 7f0fb624c640 1 --2- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 0x7f0fb0197530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0005e30 tx=0x7f0fa0030f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.859+0000 7f0faf7fe640 1 -- 172.21.15.111:0/1451324566 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fa0047070 con 0x7f0fb01027b0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.859+0000 7f0faf7fe640 1 -- 172.21.15.111:0/1451324566 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f0fa0003df0 con 0x7f0fb01027b0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.859+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb0197d60 con 0x7f0fb01027b0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.859+0000 7f0faf7fe640 1 -- 172.21.15.111:0/1451324566 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fa0033050 con 0x7f0fb01027b0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.859+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb019ce10 con 0x7f0fb01027b0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.860+0000 7f0faf7fe640 1 -- 172.21.15.111:0/1451324566 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7f0fa004a070 con 0x7f0fb01027b0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.860+0000 7f0faf7fe640 1 --2- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f0f9803f7a0 0x7f0f98041c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.860+0000 7f0faf7fe640 1 -- 172.21.15.111:0/1451324566 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0fa003e030 con 0x7f0fb01027b0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.860+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb0103a60 con 0x7f0fb01027b0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.860+0000 7f0fb5a4b640 1 --2- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f0f9803f7a0 0x7f0f98041c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:27.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.861+0000 7f0fb5a4b640 1 --2- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f0f9803f7a0 0x7f0f98041c60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40098c0 tx=0x7f0fa4006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:27.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:27.865+0000 7f0faf7fe640 1 -- 172.21.15.111:0/1451324566 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f0fa0048260 con 0x7f0fb01027b0 2024-09-16T08:54:28.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.049+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0fb0102c10 con 0x7f0f9803f7a0 2024-09-16T08:54:28.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.051+0000 7f0faf7fe640 1 -- 172.21.15.111:0/1451324566 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f0fb0102c10 con 0x7f0f9803f7a0 2024-09-16T08:54:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:54:28.051 INFO:teuthology.orchestra.run.smithi111.stdout:[{"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}] 2024-09-16T08:54:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f0f9803f7a0 msgr2=0x7f0f98041c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 --2- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f0f9803f7a0 0x7f0f98041c60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40098c0 tx=0x7f0fa4006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 msgr2=0x7f0fb0197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 --2- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 0x7f0fb0197530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0005e30 tx=0x7f0fa0030f10 comp rx=0 tx=0).stop 2024-09-16T08:54:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 shutdown_connections 2024-09-16T08:54:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 --2- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f0f9803f7a0 0x7f0f98041c60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 --2- 172.21.15.111:0/1451324566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb01027b0 0x7f0fb0197530 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 >> 172.21.15.111:0/1451324566 conn(0x7f0fb00fe220 msgr2=0x7f0fb0103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 shutdown_connections 2024-09-16T08:54:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:28.053+0000 7f0fb84b0640 1 -- 172.21.15.111:0/1451324566 wait complete. 2024-09-16T08:54:28.625 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-16T08:54:28.626 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-16T08:54:28.865 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:54:28.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:28 smithi111 ceph-mon[29803]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:28.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:28 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:29.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.323+0000 7f2cff308640 1 -- 172.21.15.111:0/495818884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 msgr2=0x7f2cf8103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:29.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 --2- 172.21.15.111:0/495818884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 0x7f2cf8103940 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4009880 tx=0x7f2cf402a5d0 comp rx=0 tx=0).stop 2024-09-16T08:54:29.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 -- 172.21.15.111:0/495818884 shutdown_connections 2024-09-16T08:54:29.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 --2- 172.21.15.111:0/495818884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 0x7f2cf8103940 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:29.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 -- 172.21.15.111:0/495818884 >> 172.21.15.111:0/495818884 conn(0x7f2cf80fd130 msgr2=0x7f2cf80ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:29.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 -- 172.21.15.111:0/495818884 shutdown_connections 2024-09-16T08:54:29.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 -- 172.21.15.111:0/495818884 wait complete. 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 Processor -- start 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 -- start start 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 0x7f2cf8197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.324+0000 7f2cff308640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf810bf00 con 0x7f2cf8103560 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cfd0a4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 0x7f2cf8197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cfd0a4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 0x7f2cf8197540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35760/0 (socket says 172.21.15.111:35760) 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cfd0a4640 1 -- 172.21.15.111:0/657053881 learned_addr learned my addr 172.21.15.111:0/657053881 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cfd0a4640 1 -- 172.21.15.111:0/657053881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf8197a80 con 0x7f2cf8103560 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cfd0a4640 1 --2- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 0x7f2cf8197540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4005ab0 tx=0x7f2cf4003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cee7fc640 1 -- 172.21.15.111:0/657053881 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cf403f070 con 0x7f2cf8103560 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cee7fc640 1 -- 172.21.15.111:0/657053881 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f2cf403bbf0 con 0x7f2cf8103560 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf8197d10 con 0x7f2cf8103560 2024-09-16T08:54:29.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cee7fc640 1 -- 172.21.15.111:0/657053881 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cf4035030 con 0x7f2cf8103560 2024-09-16T08:54:29.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.325+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf819ce20 con 0x7f2cf8103560 2024-09-16T08:54:29.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.326+0000 7f2cee7fc640 1 -- 172.21.15.111:0/657053881 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7f2cf4042070 con 0x7f2cf8103560 2024-09-16T08:54:29.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.326+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf806b4a0 con 0x7f2cf8103560 2024-09-16T08:54:29.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.326+0000 7f2cee7fc640 1 --2- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2ce403f740 0x7f2ce4041c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:29.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.326+0000 7f2cee7fc640 1 -- 172.21.15.111:0/657053881 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2cf4074b80 con 0x7f2cf8103560 2024-09-16T08:54:29.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.326+0000 7f2cfc8a3640 1 --2- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2ce403f740 0x7f2ce4041c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:29.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.327+0000 7f2cfc8a3640 1 --2- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2ce403f740 0x7f2ce4041c00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80098c0 tx=0x7f2ce8006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:29.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.331+0000 7f2cee7fc640 1 -- 172.21.15.111:0/657053881 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f2cf4031400 con 0x7f2cf8103560 2024-09-16T08:54:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.511+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f2cf81039c0 con 0x7f2cf8103560 2024-09-16T08:54:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.823+0000 7f2cee7fc640 1 -- 172.21.15.111:0/657053881 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f2cf4049060 con 0x7f2cf8103560 2024-09-16T08:54:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2024-09-16T08:54:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.825+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2ce403f740 msgr2=0x7f2ce4041c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.825+0000 7f2cff308640 1 --2- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2ce403f740 0x7f2ce4041c00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80098c0 tx=0x7f2ce8006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:29.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.825+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 msgr2=0x7f2cf8197540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:29.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.825+0000 7f2cff308640 1 --2- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 0x7f2cf8197540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4005ab0 tx=0x7f2cf4003d80 comp rx=0 tx=0).stop 2024-09-16T08:54:29.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.826+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 shutdown_connections 2024-09-16T08:54:29.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.826+0000 7f2cff308640 1 --2- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2ce403f740 0x7f2ce4041c00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:29.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.826+0000 7f2cff308640 1 --2- 172.21.15.111:0/657053881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cf8103560 0x7f2cf8197540 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:29.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.826+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 >> 172.21.15.111:0/657053881 conn(0x7f2cf80fd130 msgr2=0x7f2cf8106cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:29.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.826+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 shutdown_connections 2024-09-16T08:54:29.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:29.826+0000 7f2cff308640 1 -- 172.21.15.111:0/657053881 wait complete. 2024-09-16T08:54:30.120 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:29 smithi111 ceph-mon[29803]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T08:54:30.120 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/657053881' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-16T08:54:30.415 INFO:tasks.cephadm:Adding mon.a on smithi111 2024-09-16T08:54:30.415 INFO:tasks.cephadm:Adding mon.c on smithi111 2024-09-16T08:54:30.416 INFO:tasks.cephadm:Adding mon.b on smithi114 2024-09-16T08:54:30.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi111:172.21.15.111=a;smithi111:[v2:172.21.15.111:3301,v1:172.21.15.111:6790]=c;smithi114:172.21.15.114=b' 2024-09-16T08:54:30.733 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T08:54:30.778 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T08:54:30.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:30.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/657053881' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-16T08:54:30.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T08:54:30.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:30.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:30.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"} : dispatch 2024-09-16T08:54:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:31.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.286+0000 7fe129f89640 1 -- 172.21.15.114:0/2820739472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 msgr2=0x7fe124102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:31.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.286+0000 7fe129f89640 1 --2- 172.21.15.114:0/2820739472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 0x7fe124102cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe114009c20 tx=0x7fe11402aa90 comp rx=0 tx=0).stop 2024-09-16T08:54:31.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.286+0000 7fe129f89640 1 -- 172.21.15.114:0/2820739472 shutdown_connections 2024-09-16T08:54:31.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.286+0000 7fe129f89640 1 --2- 172.21.15.114:0/2820739472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 0x7fe124102cf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:31.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.286+0000 7fe129f89640 1 -- 172.21.15.114:0/2820739472 >> 172.21.15.114:0/2820739472 conn(0x7fe1240791f0 msgr2=0x7fe12407a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:31.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.286+0000 7fe129f89640 1 -- 172.21.15.114:0/2820739472 shutdown_connections 2024-09-16T08:54:31.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.286+0000 7fe129f89640 1 -- 172.21.15.114:0/2820739472 wait complete. 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.286+0000 7fe129f89640 1 Processor -- start 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe129f89640 1 -- start start 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe129f89640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 0x7fe1241974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe129f89640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe12410bec0 con 0x7fe124102910 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe1237fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 0x7fe1241974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe1237fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 0x7fe1241974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.114:45150/0 (socket says 172.21.15.114:45150) 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe1237fe640 1 -- 172.21.15.114:0/27390486 learned_addr learned my addr 172.21.15.114:0/27390486 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe1237fe640 1 -- 172.21.15.114:0/27390486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe124197a20 con 0x7fe124102910 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe1237fe640 1 --2- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 0x7fe1241974e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe114005e30 tx=0x7fe114030f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe120ff9640 1 -- 172.21.15.114:0/27390486 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe114047070 con 0x7fe124102910 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe124197d10 con 0x7fe124102910 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe120ff9640 1 -- 172.21.15.114:0/27390486 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fe114003df0 con 0x7fe124102910 2024-09-16T08:54:31.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.287+0000 7fe120ff9640 1 -- 172.21.15.114:0/27390486 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe114033050 con 0x7fe124102910 2024-09-16T08:54:31.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.288+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe12419cdc0 con 0x7fe124102910 2024-09-16T08:54:31.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.288+0000 7fe120ff9640 1 -- 172.21.15.114:0/27390486 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7fe11404a070 con 0x7fe124102910 2024-09-16T08:54:31.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.288+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe124107ad0 con 0x7fe124102910 2024-09-16T08:54:31.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.289+0000 7fe120ff9640 1 --2- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe10c03f740 0x7fe10c041c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:31.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.289+0000 7fe120ff9640 1 -- 172.21.15.114:0/27390486 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe11403e030 con 0x7fe124102910 2024-09-16T08:54:31.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.289+0000 7fe122ffd640 1 --2- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe10c03f740 0x7fe10c041c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:31.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.289+0000 7fe122ffd640 1 --2- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe10c03f740 0x7fe10c041c00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe1180098c0 tx=0x7fe118006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:31.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.297+0000 7fe120ff9640 1 -- 172.21.15.114:0/27390486 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fe1140373b0 con 0x7fe124102910 2024-09-16T08:54:31.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.483+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi111:172.21.15.111=a;smithi111:[v2:172.21.15.111:3301,v1:172.21.15.111:6790]=c;smithi114:172.21.15.114=b", "target": ["mon-mgr", ""]}) -- 0x7fe124198610 con 0x7fe10c03f740 2024-09-16T08:54:31.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.489+0000 7fe120ff9640 1 -- 172.21.15.114:0/27390486 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe124198610 con 0x7fe10c03f740 2024-09-16T08:54:31.491 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled mon update... 2024-09-16T08:54:31.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe10c03f740 msgr2=0x7fe10c041c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:31.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 --2- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe10c03f740 0x7fe10c041c00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe1180098c0 tx=0x7fe118006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:31.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 msgr2=0x7fe1241974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:31.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 --2- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 0x7fe1241974e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe114005e30 tx=0x7fe114030f10 comp rx=0 tx=0).stop 2024-09-16T08:54:31.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 shutdown_connections 2024-09-16T08:54:31.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 --2- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe10c03f740 0x7fe10c041c00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:31.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 --2- 172.21.15.114:0/27390486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe124102910 0x7fe1241974e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:31.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 >> 172.21.15.114:0/27390486 conn(0x7fe1240791f0 msgr2=0x7fe124104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:31.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 shutdown_connections 2024-09-16T08:54:31.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:31.491+0000 7fe129f89640 1 -- 172.21.15.114:0/27390486 wait complete. 2024-09-16T08:54:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-16T08:54:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:31.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:31.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:31.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:31.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:31 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:32.036 DEBUG:teuthology.orchestra.run.smithi111:mon.c> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.c.service 2024-09-16T08:54:32.038 DEBUG:teuthology.orchestra.run.smithi114:mon.b> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.b.service 2024-09-16T08:54:32.041 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T08:54:32.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T08:54:32.746 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:32 smithi111 ceph-mon[29803]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:32 smithi111 ceph-mon[29803]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:54:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:32 smithi111 ceph-mon[29803]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:54:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:32 smithi111 ceph-mon[29803]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi111:172.21.15.111=a;smithi111:[v2:172.21.15.111:3301,v1:172.21.15.111:6790]=c;smithi114:172.21.15.114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:32 smithi111 ceph-mon[29803]: Saving service mon spec with placement smithi111:172.21.15.111=a;smithi111:[v2:172.21.15.111:3301,v1:172.21.15.111:6790]=c;smithi114:172.21.15.114=b;count:3 2024-09-16T08:54:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:32 smithi111 ceph-mon[29803]: Deploying daemon mon.b on smithi114 2024-09-16T08:54:33.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.452+0000 7f27f0ec6640 1 -- 172.21.15.114:0/2720202902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 msgr2=0x7f27ec101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:33.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.452+0000 7f27f0ec6640 1 --2- 172.21.15.114:0/2720202902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 0x7f27ec101640 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009c20 tx=0x7f27dc02ab20 comp rx=0 tx=0).stop 2024-09-16T08:54:33.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.452+0000 7f27f0ec6640 1 -- 172.21.15.114:0/2720202902 shutdown_connections 2024-09-16T08:54:33.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.452+0000 7f27f0ec6640 1 --2- 172.21.15.114:0/2720202902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 0x7f27ec101640 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:33.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.452+0000 7f27f0ec6640 1 -- 172.21.15.114:0/2720202902 >> 172.21.15.114:0/2720202902 conn(0x7f27ec0fd130 msgr2=0x7f27ec0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:33.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.452+0000 7f27f0ec6640 1 -- 172.21.15.114:0/2720202902 shutdown_connections 2024-09-16T08:54:33.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.452+0000 7f27f0ec6640 1 -- 172.21.15.114:0/2720202902 wait complete. 2024-09-16T08:54:33.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.453+0000 7f27f0ec6640 1 Processor -- start 2024-09-16T08:54:33.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.453+0000 7f27f0ec6640 1 -- start start 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.453+0000 7f27f0ec6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 0x7f27ec199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.453+0000 7f27f0ec6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f27ec10e130 con 0x7f27ec101260 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.453+0000 7f27ea59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 0x7f27ec199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.453+0000 7f27ea59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 0x7f27ec199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.114:45162/0 (socket says 172.21.15.114:45162) 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.453+0000 7f27ea59c640 1 -- 172.21.15.114:0/3581879168 learned_addr learned my addr 172.21.15.114:0/3581879168 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.454+0000 7f27ea59c640 1 -- 172.21.15.114:0/3581879168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27ec199c50 con 0x7f27ec101260 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.454+0000 7f27ea59c640 1 --2- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 0x7f27ec199710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009430 tx=0x7f27dc003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.454+0000 7f27db7fe640 1 -- 172.21.15.114:0/3581879168 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27dc047070 con 0x7f27ec101260 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.454+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27ec199ee0 con 0x7f27ec101260 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.454+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27ec19eff0 con 0x7f27ec101260 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.454+0000 7f27db7fe640 1 -- 172.21.15.114:0/3581879168 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f27dc02eab0 con 0x7f27ec101260 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.454+0000 7f27db7fe640 1 -- 172.21.15.114:0/3581879168 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27dc004230 con 0x7f27ec101260 2024-09-16T08:54:33.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.455+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27cc0050b0 con 0x7f27ec101260 2024-09-16T08:54:33.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.455+0000 7f27db7fe640 1 -- 172.21.15.114:0/3581879168 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7f27dc04a070 con 0x7f27ec101260 2024-09-16T08:54:33.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.456+0000 7f27db7fe640 1 --2- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f27d403f740 0x7f27d4041c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:33.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.456+0000 7f27db7fe640 1 -- 172.21.15.114:0/3581879168 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f27dc03d030 con 0x7f27ec101260 2024-09-16T08:54:33.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.456+0000 7f27e9d9b640 1 --2- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f27d403f740 0x7f27d4041c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:33.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.456+0000 7f27e9d9b640 1 --2- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f27d403f740 0x7f27d4041c00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f27e00098c0 tx=0x7f27e0006db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:33.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.460+0000 7f27db7fe640 1 -- 172.21.15.114:0/3581879168 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f27dc032030 con 0x7f27ec101260 2024-09-16T08:54:33.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.728+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f27cc0053a0 con 0x7f27ec101260 2024-09-16T08:54:33.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.729+0000 7f27db7fe640 1 -- 172.21.15.114:0/3581879168 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f27dc04ac70 con 0x7f27ec101260 2024-09-16T08:54:33.730 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:54:33.731 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"f774c998-7408-11ef-bceb-c7b262605968","modified":"2024-09-16T08:53:05.472892Z","created":"2024-09-16T08:53:05.472892Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-16T08:54:33.731 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-09-16T08:54:33.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f27d403f740 msgr2=0x7f27d4041c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:33.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 --2- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f27d403f740 0x7f27d4041c00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f27e00098c0 tx=0x7f27e0006db0 comp rx=0 tx=0).stop 2024-09-16T08:54:33.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 msgr2=0x7f27ec199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:33.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 --2- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 0x7f27ec199710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009430 tx=0x7f27dc003f80 comp rx=0 tx=0).stop 2024-09-16T08:54:33.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 shutdown_connections 2024-09-16T08:54:33.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 --2- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f27d403f740 0x7f27d4041c00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:33.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 --2- 172.21.15.114:0/3581879168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec101260 0x7f27ec199710 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:33.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 >> 172.21.15.114:0/3581879168 conn(0x7f27ec0fd130 msgr2=0x7f27ec0fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:33.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 shutdown_connections 2024-09-16T08:54:33.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:33.731+0000 7f27f0ec6640 1 -- 172.21.15.114:0/3581879168 wait complete. 2024-09-16T08:54:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/3581879168' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T08:54:35.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:34 smithi111 ceph-mon[29803]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:35.932 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T08:54:35.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T08:54:36.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 podman[33251]: 2024-09-16 08:54:36.136725908 +0000 UTC m=+0.120965321 container create 4c26029b2e1217e676f80bae947be0b0cdaee9666704adcd2208563666737271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-b, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-16T08:54:36.434 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 podman[33251]: 2024-09-16 08:54:36.322355785 +0000 UTC m=+0.306595199 container init 4c26029b2e1217e676f80bae947be0b0cdaee9666704adcd2208563666737271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-13-1248-8293d73) 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 podman[33251]: 2024-09-16 08:54:36.327098761 +0000 UTC m=+0.311338166 container start 4c26029b2e1217e676f80bae947be0b0cdaee9666704adcd2208563666737271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: set uid:gid to 167:167 (ceph:ceph) 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: ceph version 19.3.0-4917-g8293d73f (8293d73f8690540e843a81caec373f9cc29cf705) squid (dev), process ceph-mon, pid 7 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: pidfile_write: ignore empty --pid-file 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: load: jerasure load: lrc 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: RocksDB version: 7.9.2 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Git sha 0 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Compile date 2024-09-13 14:50:41 2024-09-16T08:54:36.628 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: DB SUMMARY 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: DB Session ID: B1SX0A3KWTLXBMAB1X1E 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: CURRENT file: CURRENT 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: IDENTITY file: IDENTITY 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.error_if_exists: 0 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.create_if_missing: 0 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.paranoid_checks: 1 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.env: 0x56197abf6120 2024-09-16T08:54:36.629 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.fs: PosixFileSystem 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.info_log: 0x56197b703700 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_file_opening_threads: 16 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.statistics: (nil) 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.use_fsync: 0 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_log_file_size: 0 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.keep_log_file_num: 1000 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.recycle_log_file_num: 0 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.allow_fallocate: 1 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.allow_mmap_reads: 0 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.allow_mmap_writes: 0 2024-09-16T08:54:36.630 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.use_direct_reads: 0 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.create_missing_column_families: 0 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.db_log_dir: 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.wal_dir: 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.advise_random_on_open: 1 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.db_write_buffer_size: 0 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.write_buffer_manager: 0x56197b707b80 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-16T08:54:36.631 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.rate_limiter: (nil) 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.wal_recovery_mode: 2 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.enable_thread_tracking: 0 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.enable_pipelined_write: 0 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.unordered_write: 0 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.row_cache: None 2024-09-16T08:54:36.632 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.wal_filter: None 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.allow_ingest_behind: 0 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.two_write_queues: 0 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.manual_wal_flush: 0 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.wal_compression: 0 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.atomic_flush: 0 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-16T08:54:36.634 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.log_readahead_size: 0 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.best_efforts_recovery: 0 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.allow_data_in_errors: 0 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.db_host_id: __hostname__ 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_background_jobs: 2 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_background_compactions: -1 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_subcompactions: 1 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-16T08:54:36.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_total_wal_size: 0 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_open_files: -1 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bytes_per_sync: 0 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_readahead_size: 0 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_background_flushes: -1 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Compression algorithms supported: 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: kZSTD supported: 0 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: kXpressCompression supported: 0 2024-09-16T08:54:36.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: kBZip2Compression supported: 0 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: kLZ4Compression supported: 1 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: kZlibCompression supported: 1 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: kLZ4HCCompression supported: 1 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: kSnappyCompression supported: 1 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.merge_operator: 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_filter: None 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_filter_factory: None 2024-09-16T08:54:36.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.sst_partitioner_factory: None 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56197b703320) 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_type: 0 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_index_type: 0 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_shortening: 1 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: checksum: 4 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: no_block_cache: 0 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache: 0x56197b6f7610 2024-09-16T08:54:36.638 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_options: 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: capacity : 536870912 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: num_shard_bits : 4 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: strict_capacity_limit : 0 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_compressed: (nil) 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: persistent_cache: (nil) 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size: 4096 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size_deviation: 10 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_restart_interval: 16 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_block_restart_interval: 1 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: metadata_block_size: 4096 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: partition_filters: 0 2024-09-16T08:54:36.639 INFO:journalctl@ceph.mon.b.smithi114.stdout: use_delta_encoding: 1 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: filter_policy: bloomfilter 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: whole_key_filtering: 1 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: verify_compression: 0 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: format_version: 5 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_index_compression: 1 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_align: 0 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: max_auto_readahead_size: 262144 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: prepopulate_block_cache: 0 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: initial_auto_readahead_size: 8192 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.write_buffer_size: 33554432 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_write_buffer_number: 2 2024-09-16T08:54:36.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression: NoCompression 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression: Disabled 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.prefix_extractor: nullptr 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.num_levels: 7 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-16T08:54:36.641 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression_opts.level: 32767 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression_opts.strategy: 0 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression_opts.enabled: false 2024-09-16T08:54:36.642 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.target_file_size_base: 67108864 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-16T08:54:36.643 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.arena_block_size: 1048576 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.disable_auto_compactions: 0 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-16T08:54:36.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.inplace_update_support: 0 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.bloom_locality: 0 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.max_successive_merges: 0 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.paranoid_file_checks: 0 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.force_consistency_checks: 1 2024-09-16T08:54:36.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.report_bg_io_stats: 0 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.ttl: 2592000 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.enable_blob_files: false 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.min_blob_size: 0 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.blob_file_size: 268435456 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.blob_file_starting_level: 0 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-16T08:54:36.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 01fd0a91-cdc4-4bb5-b8e5-81f5f0efbe8c 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726476876353835, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726476876354220, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726476876, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "01fd0a91-cdc4-4bb5-b8e5-81f5f0efbe8c", "db_session_id": "B1SX0A3KWTLXBMAB1X1E", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726476876354315, "job": 1, "event": "recovery_finished"} 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56197b726e00 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: DB pointer 0x56197b828000 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** DB Stats ** 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T08:54:36.647 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-09-16T08:54:36.648 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T08:54:36.649 INFO:journalctl@ceph.mon.b.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache BinnedLRUCache@0x56197b6f7610#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: using public_addr v2:172.21.15.114:0/0 -> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: starting mon.b rank -1 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(???) e0 preinit fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).mds e1 new map 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout: e1 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout: btime 2024-09-16T08:53:08:312988+0000 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T08:54:36.650 INFO:journalctl@ceph.mon.b.smithi114.stdout: legacy client fscid: -1 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout: No filesystems configured 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/657053881' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/657053881' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"} : dispatch 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:36.652 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi111:172.21.15.111=a;smithi111:[v2:172.21.15.111:3301,v1:172.21.15.111:6790]=c;smithi114:172.21.15.114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: Saving service mon spec with placement smithi111:172.21.15.111=a;smithi111:[v2:172.21.15.111:3301,v1:172.21.15.111:6790]=c;smithi114:172.21.15.114=b;count:3 2024-09-16T08:54:36.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: Deploying daemon mon.b on smithi114 2024-09-16T08:54:36.654 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/3581879168' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T08:54:36.654 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:36.654 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 ceph-mon[33315]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-16T08:54:36.654 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 bash[33251]: 4c26029b2e1217e676f80bae947be0b0cdaee9666704adcd2208563666737271 2024-09-16T08:54:36.654 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:36 smithi114 systemd[1]: Started Ceph mon.b for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:54:41.724 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.724 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:54:41.724 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: mon.a calling monitor election 2024-09-16T08:54:41.724 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: mon.b calling monitor election 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: monmap epoch 2 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:41.725 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: last_changed 2024-09-16T08:54:36.379243+0000 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: created 2024-09-16T08:53:05.472892+0000 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: min_mon_release 19 (squid) 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: election_strategy: 1 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: 0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: fsmap 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: mgrmap e13: y(active, since 34s) 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: overall HEALTH_OK 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:41.726 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: mon.a calling monitor election 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: mon.b calling monitor election 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: monmap epoch 2 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: last_changed 2024-09-16T08:54:36.379243+0000 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: created 2024-09-16T08:53:05.472892+0000 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: min_mon_release 19 (squid) 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: election_strategy: 1 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: 0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: fsmap 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: mgrmap e13: y(active, since 34s) 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: overall HEALTH_OK 2024-09-16T08:54:41.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:41.916 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:41.916 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:41.916 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:41.916 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:42 smithi111 ceph-mon[29803]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:42 smithi111 ceph-mon[29803]: Deploying daemon mon.c on smithi111 2024-09-16T08:54:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:42 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:42.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.841+0000 7fb5d1142640 1 -- 172.21.15.114:0/3406750836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 msgr2=0x7fb5b400a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:42.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.841+0000 7fb5d1142640 1 --2- 172.21.15.114:0/3406750836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 0x7fb5b400a890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc1027c0 tx=0x7fb5bc02c4a0 comp rx=0 tx=0).stop 2024-09-16T08:54:42.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.841+0000 7fb5d1142640 1 -- 172.21.15.114:0/3406750836 shutdown_connections 2024-09-16T08:54:42.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.841+0000 7fb5d1142640 1 --2- 172.21.15.114:0/3406750836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 0x7fb5b400a890 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:42.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.841+0000 7fb5d1142640 1 -- 172.21.15.114:0/3406750836 >> 172.21.15.114:0/3406750836 conn(0x7fb5cc0fd220 msgr2=0x7fb5cc0ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:42.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.841+0000 7fb5d1142640 1 -- 172.21.15.114:0/3406750836 shutdown_connections 2024-09-16T08:54:42.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.841+0000 7fb5d1142640 1 -- 172.21.15.114:0/3406750836 wait complete. 2024-09-16T08:54:42.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:42 smithi114 ceph-mon[33315]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:42.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:42 smithi114 ceph-mon[33315]: Deploying daemon mon.c on smithi111 2024-09-16T08:54:42.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:42 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:42.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.841+0000 7fb5d1142640 1 Processor -- start 2024-09-16T08:54:42.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.841+0000 7fb5d1142640 1 -- start start 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5d1142640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 0x7fb5cc1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5d1142640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc199e30 0x7fb5cc19f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5d1142640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc06b5d0 con 0x7fb5b4008420 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5d1142640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc06b450 con 0x7fb5cc199e30 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5cad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 0x7fb5cc1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5ca59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc199e30 0x7fb5cc19f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5cad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 0x7fb5cc1998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.114:41752/0 (socket says 172.21.15.114:41752) 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5cad9d640 1 -- 172.21.15.114:0/2113754520 learned_addr learned my addr 172.21.15.114:0/2113754520 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5ca59c640 1 -- 172.21.15.114:0/2113754520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc199e30 msgr2=0x7fb5cc19f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5ca59c640 1 -- 172.21.15.114:0/2113754520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc199e30 msgr2=0x7fb5cc19f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T08:54:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5ca59c640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc199e30 0x7fb5cc19f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5ca59c640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc199e30 0x7fb5cc19f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5cad9d640 1 -- 172.21.15.114:0/2113754520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc199e30 msgr2=0x7fb5cc19f1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5cad9d640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc199e30 0x7fb5cc19f1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5cad9d640 1 -- 172.21.15.114:0/2113754520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc19a4b0 con 0x7fb5b4008420 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.842+0000 7fb5cad9d640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 0x7fb5cc1998f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc036040 tx=0x7fb5bc0336a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.843+0000 7fb5b3fff640 1 -- 172.21.15.114:0/2113754520 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb5bc038040 con 0x7fb5b4008420 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.843+0000 7fb5b3fff640 1 -- 172.21.15.114:0/2113754520 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fb5bc032690 con 0x7fb5b4008420 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.843+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5cc19f710 con 0x7fb5b4008420 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.843+0000 7fb5b3fff640 1 -- 172.21.15.114:0/2113754520 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb5bc004820 con 0x7fb5b4008420 2024-09-16T08:54:42.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.843+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5cc19fcd0 con 0x7fb5b4008420 2024-09-16T08:54:42.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.844+0000 7fb5b3fff640 1 -- 172.21.15.114:0/2113754520 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7fb5bc044020 con 0x7fb5b4008420 2024-09-16T08:54:42.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.844+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5cc102a40 con 0x7fb5b4008420 2024-09-16T08:54:42.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.844+0000 7fb5b3fff640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb5b803f530 0x7fb5b80419f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:42.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.844+0000 7fb5b3fff640 1 -- 172.21.15.114:0/2113754520 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb5bc076680 con 0x7fb5b4008420 2024-09-16T08:54:42.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.845+0000 7fb5ca59c640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb5b803f530 0x7fb5b80419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:42.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.845+0000 7fb5ca59c640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb5b803f530 0x7fb5b80419f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0006fd0 tx=0x7fb5c0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:42.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:42.850+0000 7fb5b3fff640 1 -- 172.21.15.114:0/2113754520 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fb5bc04b020 con 0x7fb5b4008420 2024-09-16T08:54:43.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.096+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb5cc19aac0 con 0x7fb5b4008420 2024-09-16T08:54:43.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.097+0000 7fb5b3fff640 1 -- 172.21.15.114:0/2113754520 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7fb5bc0453b0 con 0x7fb5b4008420 2024-09-16T08:54:43.098 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:54:43.098 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":2,"fsid":"f774c998-7408-11ef-bceb-c7b262605968","modified":"2024-09-16T08:54:36.379243Z","created":"2024-09-16T08:53:05.472892Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T08:54:43.098 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 2 2024-09-16T08:54:43.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb5b803f530 msgr2=0x7fb5b80419f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:43.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb5b803f530 0x7fb5b80419f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0006fd0 tx=0x7fb5c0008040 comp rx=0 tx=0).stop 2024-09-16T08:54:43.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 msgr2=0x7fb5cc1998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:43.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 0x7fb5cc1998f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc036040 tx=0x7fb5bc0336a0 comp rx=0 tx=0).stop 2024-09-16T08:54:43.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 shutdown_connections 2024-09-16T08:54:43.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb5b803f530 0x7fb5b80419f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:43.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc199e30 0x7fb5cc19f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:43.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 --2- 172.21.15.114:0/2113754520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b4008420 0x7fb5cc1998f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:43.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 >> 172.21.15.114:0/2113754520 conn(0x7fb5cc0fd220 msgr2=0x7fb5cc102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:43.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 shutdown_connections 2024-09-16T08:54:43.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:43.099+0000 7fb5d1142640 1 -- 172.21.15.114:0/2113754520 wait complete. 2024-09-16T08:54:43.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/2113754520' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T08:54:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/2113754520' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T08:54:44.335 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 systemd[1]: Starting Ceph mon.c for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T08:54:44.585 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[29803]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.585 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 podman[40018]: 2024-09-16 08:54:44.334733134 +0000 UTC m=+0.107105992 container create 86f0e47425e123e3f3dd2975809f219802435986def775d475f6fee739970295 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T08:54:44.585 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 podman[40018]: 2024-09-16 08:54:44.243040698 +0000 UTC m=+0.015413569 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T08:54:44.585 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 podman[40018]: 2024-09-16 08:54:44.553698301 +0000 UTC m=+0.326071159 container init 86f0e47425e123e3f3dd2975809f219802435986def775d475f6fee739970295 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-c, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 podman[40018]: 2024-09-16 08:54:44.558381631 +0000 UTC m=+0.330754490 container start 86f0e47425e123e3f3dd2975809f219802435986def775d475f6fee739970295 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: set uid:gid to 167:167 (ceph:ceph) 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: ceph version 19.3.0-4917-g8293d73f (8293d73f8690540e843a81caec373f9cc29cf705) squid (dev), process ceph-mon, pid 7 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: pidfile_write: ignore empty --pid-file 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: load: jerasure load: lrc 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: RocksDB version: 7.9.2 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Git sha 0 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Compile date 2024-09-13 14:50:41 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: DB SUMMARY 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: DB Session ID: W4ZBLRCIY4XVXNR3PLO1 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: CURRENT file: CURRENT 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: IDENTITY file: IDENTITY 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-16T08:54:44.586 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-16T08:54:44.587 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-16T08:54:44.587 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.error_if_exists: 0 2024-09-16T08:54:44.587 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.create_if_missing: 0 2024-09-16T08:54:44.587 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.paranoid_checks: 1 2024-09-16T08:54:44.587 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-16T08:54:44.587 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-16T08:54:44.587 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.env: 0x5581d09d7120 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.fs: PosixFileSystem 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.info_log: 0x5581d3081700 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_file_opening_threads: 16 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.statistics: (nil) 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.use_fsync: 0 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_log_file_size: 0 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.keep_log_file_num: 1000 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.recycle_log_file_num: 0 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.allow_fallocate: 1 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.allow_mmap_reads: 0 2024-09-16T08:54:44.588 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.allow_mmap_writes: 0 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.use_direct_reads: 0 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.create_missing_column_families: 0 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.db_log_dir: 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.wal_dir: 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.advise_random_on_open: 1 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.db_write_buffer_size: 0 2024-09-16T08:54:44.589 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.write_buffer_manager: 0x5581d3085b80 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.rate_limiter: (nil) 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.wal_recovery_mode: 2 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.enable_thread_tracking: 0 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.enable_pipelined_write: 0 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.unordered_write: 0 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-16T08:54:44.590 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.row_cache: None 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.wal_filter: None 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.allow_ingest_behind: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.two_write_queues: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.manual_wal_flush: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.wal_compression: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.atomic_flush: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.log_readahead_size: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.best_efforts_recovery: 0 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-16T08:54:44.591 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.allow_data_in_errors: 0 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.db_host_id: __hostname__ 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_background_jobs: 2 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_background_compactions: -1 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_subcompactions: 1 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_total_wal_size: 0 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-16T08:54:44.592 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_open_files: -1 2024-09-16T08:54:44.593 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bytes_per_sync: 0 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_readahead_size: 0 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_background_flushes: -1 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Compression algorithms supported: 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: kZSTD supported: 0 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: kXpressCompression supported: 0 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: kBZip2Compression supported: 0 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-16T08:54:44.594 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: kLZ4Compression supported: 1 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: kZlibCompression supported: 1 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: kLZ4HCCompression supported: 1 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: kSnappyCompression supported: 1 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.merge_operator: 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_filter: None 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_filter_factory: None 2024-09-16T08:54:44.595 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.sst_partitioner_factory: None 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5581d3081320) 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: cache_index_and_filter_blocks: 1 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: pin_top_level_index_and_filter: 1 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: index_type: 0 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: data_block_index_type: 0 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: index_shortening: 1 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: checksum: 4 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: no_block_cache: 0 2024-09-16T08:54:44.596 INFO:journalctl@ceph.mon.c.smithi111.stdout: block_cache: 0x5581d3075610 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: block_cache_name: BinnedLRUCache 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: block_cache_options: 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: capacity : 536870912 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: num_shard_bits : 4 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: strict_capacity_limit : 0 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: high_pri_pool_ratio: 0.000 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: block_cache_compressed: (nil) 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: persistent_cache: (nil) 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: block_size: 4096 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: block_size_deviation: 10 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: block_restart_interval: 16 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: index_block_restart_interval: 1 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: metadata_block_size: 4096 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: partition_filters: 0 2024-09-16T08:54:44.597 INFO:journalctl@ceph.mon.c.smithi111.stdout: use_delta_encoding: 1 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: filter_policy: bloomfilter 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: whole_key_filtering: 1 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: verify_compression: 0 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: read_amp_bytes_per_bit: 0 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: format_version: 5 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: enable_index_compression: 1 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: block_align: 0 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: max_auto_readahead_size: 262144 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: prepopulate_block_cache: 0 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: initial_auto_readahead_size: 8192 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout: num_file_reads_for_auto_readahead: 2 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.write_buffer_size: 33554432 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_write_buffer_number: 2 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression: NoCompression 2024-09-16T08:54:44.598 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression: Disabled 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.prefix_extractor: nullptr 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.num_levels: 7 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-16T08:54:44.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression_opts.level: 32767 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression_opts.strategy: 0 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression_opts.enabled: false 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.target_file_size_base: 67108864 2024-09-16T08:54:44.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.arena_block_size: 1048576 2024-09-16T08:54:44.601 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.disable_auto_compactions: 0 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.inplace_update_support: 0 2024-09-16T08:54:44.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.bloom_locality: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.max_successive_merges: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.paranoid_file_checks: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.force_consistency_checks: 1 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.report_bg_io_stats: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.ttl: 2592000 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-16T08:54:44.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.enable_blob_files: false 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.min_blob_size: 0 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.blob_file_size: 268435456 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.blob_file_starting_level: 0 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ca23d0f3-cb2a-4596-9897-4dd76cbc3eda 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726476884585011, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-16T08:54:44.604 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-16T08:54:44.605 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726476884585440, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726476884, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ca23d0f3-cb2a-4596-9897-4dd76cbc3eda", "db_session_id": "W4ZBLRCIY4XVXNR3PLO1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-16T08:54:44.605 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726476884585607, "job": 1, "event": "recovery_finished"} 2024-09-16T08:54:44.605 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-16T08:54:44.605 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-16T08:54:44.605 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5581d30a4e00 2024-09-16T08:54:44.605 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: DB pointer 0x5581d31a6000 2024-09-16T08:54:44.605 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-16T08:54:44.605 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: ** DB Stats ** 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: ** Compaction Stats [default] ** 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-16T08:54:44.606 INFO:journalctl@ceph.mon.c.smithi111.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: ** Compaction Stats [default] ** 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-16T08:54:44.607 INFO:journalctl@ceph.mon.c.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout: Cumulative compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout: Interval compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout: Block cache BinnedLRUCache@0x5581d3075610#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout: 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: using public_addrv [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: starting mon.c rank -1 at public addrs [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] at bind addrs [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:44.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(???) e0 preinit fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:44.624 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T08:54:44.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T08:54:44.876 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:54:44.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:44 smithi114 ceph-mon[33315]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).mds e1 new map 2024-09-16T08:54:44.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-16T08:54:44.926 INFO:journalctl@ceph.mon.c.smithi111.stdout: e1 2024-09-16T08:54:44.926 INFO:journalctl@ceph.mon.c.smithi111.stdout: btime 2024-09-16T08:53:08:312988+0000 2024-09-16T08:54:44.926 INFO:journalctl@ceph.mon.c.smithi111.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T08:54:44.926 INFO:journalctl@ceph.mon.c.smithi111.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T08:54:44.926 INFO:journalctl@ceph.mon.c.smithi111.stdout: legacy client fscid: -1 2024-09-16T08:54:44.926 INFO:journalctl@ceph.mon.c.smithi111.stdout: 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout: No filesystems configured 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/657053881' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/657053881' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"} : dispatch 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:44.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:44.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-16T08:54:44.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:44.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:44.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:44.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi111:172.21.15.111=a;smithi111:[v2:172.21.15.111:3301,v1:172.21.15.111:6790]=c;smithi114:172.21.15.114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: Saving service mon spec with placement smithi111:172.21.15.111=a;smithi111:[v2:172.21.15.111:3301,v1:172.21.15.111:6790]=c;smithi114:172.21.15.114=b;count:3 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: Deploying daemon mon.b on smithi114 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/3581879168' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T08:54:44.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.a calling monitor election 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.b calling monitor election 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:44.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: monmap epoch 2 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: last_changed 2024-09-16T08:54:36.379243+0000 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: created 2024-09-16T08:53:05.472892+0000 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: min_mon_release 19 (squid) 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: election_strategy: 1 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: 0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T08:54:44.932 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: fsmap 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mgrmap e13: y(active, since 34s) 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: overall HEALTH_OK 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: Deploying daemon mon.c on smithi111 2024-09-16T08:54:44.933 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:44.934 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/2113754520' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T08:54:44.934 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:44.934 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 ceph-mon[40056]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-16T08:54:44.934 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 bash[40018]: 86f0e47425e123e3f3dd2975809f219802435986def775d475f6fee739970295 2024-09-16T08:54:44.934 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:44 smithi111 systemd[1]: Started Ceph mon.c for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:54:49.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.759+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3731795021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 msgr2=0x7f2a940059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:49.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.759+0000 7f2aaf36b640 1 --2- 172.21.15.114:0/3731795021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 0x7f2a940059c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8101500 tx=0x7f2aa402c490 comp rx=0 tx=0).stop 2024-09-16T08:54:49.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.759+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3731795021 shutdown_connections 2024-09-16T08:54:49.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.759+0000 7f2aaf36b640 1 --2- 172.21.15.114:0/3731795021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 0x7f2a940059c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:49.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.759+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3731795021 >> 172.21.15.114:0/3731795021 conn(0x7f2aa80fd220 msgr2=0x7f2aa80ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:49.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.759+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3731795021 shutdown_connections 2024-09-16T08:54:49.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.759+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3731795021 wait complete. 2024-09-16T08:54:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.760+0000 7f2aaf36b640 1 Processor -- start 2024-09-16T08:54:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.760+0000 7f2aaf36b640 1 -- start start 2024-09-16T08:54:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.760+0000 7f2aaf36b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 0x7f2aa818e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.760+0000 7f2aaf36b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 0x7f2aa818ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.760+0000 7f2aaf36b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa818f3e0 0x7f2aa81a9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.760+0000 7f2aaf36b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa8108960 con 0x7f2a94003530 2024-09-16T08:54:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.760+0000 7f2aaf36b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa81087e0 con 0x7f2aa8192680 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.760+0000 7f2aaf36b640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f2aa8108ae0 con 0x7f2aa818f3e0 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aac906640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 0x7f2aa818ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad107640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 0x7f2aa818e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad107640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 0x7f2aa818e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.114:41850/0 (socket says 172.21.15.114:41850) 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad107640 1 -- 172.21.15.114:0/3293061321 learned_addr learned my addr 172.21.15.114:0/3293061321 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aac906640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 0x7f2aa818ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46506/0 (socket says 172.21.15.114:46506) 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad908640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa818f3e0 0x7f2aa81a9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aac906640 1 -- 172.21.15.114:0/3293061321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 msgr2=0x7f2aa818ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aac906640 1 -- 172.21.15.114:0/3293061321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 msgr2=0x7f2aa818ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aac906640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 0x7f2aa818ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T08:54:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad908640 1 -- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa818f3e0 msgr2=0x7f2aa81a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad908640 1 -- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa818f3e0 msgr2=0x7f2aa81a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad908640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa818f3e0 0x7f2aa81a9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad908640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa818f3e0 0x7f2aa81a9f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aac906640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 0x7f2aa818ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad107640 1 -- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa818f3e0 msgr2=0x7f2aa81a9f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad107640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa818f3e0 0x7f2aa81a9f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad107640 1 -- 172.21.15.114:0/3293061321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 msgr2=0x7f2aa818ee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad107640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 0x7f2aa818ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad107640 1 -- 172.21.15.114:0/3293061321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa81aa750 con 0x7f2a94003530 2024-09-16T08:54:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aad107640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 0x7f2aa818e8c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2aa402ca00 tx=0x7f2aa4034e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:49.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2a9e7fc640 1 -- 172.21.15.114:0/3293061321 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4040020 con 0x7f2a94003530 2024-09-16T08:54:49.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2a9e7fc640 1 -- 172.21.15.114:0/3293061321 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f2aa403c410 con 0x7f2a94003530 2024-09-16T08:54:49.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aa81aa9e0 con 0x7f2a94003530 2024-09-16T08:54:49.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2a9e7fc640 1 -- 172.21.15.114:0/3293061321 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4038040 con 0x7f2a94003530 2024-09-16T08:54:49.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.761+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aa81aaf70 con 0x7f2a94003530 2024-09-16T08:54:49.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.762+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aa8102a40 con 0x7f2a94003530 2024-09-16T08:54:49.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.762+0000 7f2a9e7fc640 1 -- 172.21.15.114:0/3293061321 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7f2aa4043020 con 0x7f2a94003530 2024-09-16T08:54:49.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.763+0000 7f2a9e7fc640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2a8c03fc00 0x7f2a8c0420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:49.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.763+0000 7f2a9e7fc640 1 -- 172.21.15.114:0/3293061321 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2aa4076380 con 0x7f2a94003530 2024-09-16T08:54:49.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.763+0000 7f2aac906640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2a8c03fc00 0x7f2a8c0420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:49.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.763+0000 7f2aac906640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2a8c03fc00 0x7f2a8c0420c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a98006fd0 tx=0x7f2a98008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:49.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:49.767+0000 7f2a9e7fc640 1 -- 172.21.15.114:0/3293061321 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f2aa404a020 con 0x7f2a94003530 2024-09-16T08:54:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:54:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: mon.a calling monitor election 2024-09-16T08:54:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: mon.b calling monitor election 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: mon.c calling monitor election 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: monmap epoch 3 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: last_changed 2024-09-16T08:54:44.610060+0000 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: created 2024-09-16T08:53:05.472892+0000 2024-09-16T08:54:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: min_mon_release 19 (squid) 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: election_strategy: 1 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: 0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: 2: [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] mon.c 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: fsmap 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: mgrmap e13: y(active, since 42s) 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: overall HEALTH_OK 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: mon.a calling monitor election 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: mon.b calling monitor election 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: mon.c calling monitor election 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:49.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: monmap epoch 3 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: last_changed 2024-09-16T08:54:44.610060+0000 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: created 2024-09-16T08:53:05.472892+0000 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: min_mon_release 19 (squid) 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: election_strategy: 1 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: 0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: 2: [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] mon.c 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: fsmap 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: mgrmap e13: y(active, since 42s) 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: overall HEALTH_OK 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:49.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:49.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:49.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:50.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.039+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2aa81ab1f0 con 0x7f2a94003530 2024-09-16T08:54:50.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.039+0000 7f2a9e7fc640 1 -- 172.21.15.114:0/3293061321 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1289 (secure 0 0 0) 0x7f2aa4040d50 con 0x7f2a94003530 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: mon.a calling monitor election 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: mon.b calling monitor election 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: mon.c calling monitor election 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:50.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: monmap epoch 3 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: fsid f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: last_changed 2024-09-16T08:54:44.610060+0000 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: created 2024-09-16T08:53:05.472892+0000 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: min_mon_release 19 (squid) 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: election_strategy: 1 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: 0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: 2: [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] mon.c 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: fsmap 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: mgrmap e13: y(active, since 42s) 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: overall HEALTH_OK 2024-09-16T08:54:50.042 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:50.043 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:50.043 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:50.043 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:50.043 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:54:50.043 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":3,"fsid":"f774c998-7408-11ef-bceb-c7b262605968","modified":"2024-09-16T08:54:44.610060Z","created":"2024-09-16T08:53:05.472892Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3301","nonce":0},{"type":"v1","addr":"172.21.15.111:6790","nonce":0}]},"addr":"172.21.15.111:6790/0","public_addr":"172.21.15.111:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-16T08:54:50.044 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 3 2024-09-16T08:54:50.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2a8c03fc00 msgr2=0x7f2a8c0420c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:50.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2a8c03fc00 0x7f2a8c0420c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a98006fd0 tx=0x7f2a98008040 comp rx=0 tx=0).stop 2024-09-16T08:54:50.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 msgr2=0x7f2aa818e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:50.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 0x7f2aa818e8c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2aa402ca00 tx=0x7f2aa4034e50 comp rx=0 tx=0).stop 2024-09-16T08:54:50.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 shutdown_connections 2024-09-16T08:54:50.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f2a8c03fc00 0x7f2a8c0420c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:50.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa818f3e0 0x7f2aa81a9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:50.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa8192680 0x7f2aa818ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:50.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 --2- 172.21.15.114:0/3293061321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a94003530 0x7f2aa818e8c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:50.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 >> 172.21.15.114:0/3293061321 conn(0x7f2aa80fd220 msgr2=0x7f2aa806b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:50.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 shutdown_connections 2024-09-16T08:54:50.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:50.042+0000 7f2aaf36b640 1 -- 172.21.15.114:0/3293061321 wait complete. 2024-09-16T08:54:50.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:50 smithi114 ceph-mon[33315]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:50.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:50.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/3293061321' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T08:54:50.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:50 smithi111 ceph-mon[29803]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/3293061321' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T08:54:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:50.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:50 smithi111 ceph-mon[40056]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:50.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:50.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/3293061321' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-16T08:54:50.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:54:51.348 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-16T08:54:51.348 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-16T08:54:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T08:54:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:51.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:51.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T08:54:51.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:52.066 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:54:52.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T08:54:52.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.843+0000 7fe1dc285640 1 -- 172.21.15.111:0/2960461715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d41016c0 msgr2=0x7fe1d4101aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.843+0000 7fe1dc285640 1 --2- 172.21.15.111:0/2960461715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d41016c0 0x7fe1d4101aa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009880 tx=0x7fe1d002a5d0 comp rx=0 tx=0).stop 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.843+0000 7fe1dc285640 1 -- 172.21.15.111:0/2960461715 shutdown_connections 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.843+0000 7fe1dc285640 1 --2- 172.21.15.111:0/2960461715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d41016c0 0x7fe1d4101aa0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.843+0000 7fe1dc285640 1 -- 172.21.15.111:0/2960461715 >> 172.21.15.111:0/2960461715 conn(0x7fe1d40fd130 msgr2=0x7fe1d40ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.843+0000 7fe1dc285640 1 -- 172.21.15.111:0/2960461715 shutdown_connections 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.843+0000 7fe1dc285640 1 -- 172.21.15.111:0/2960461715 wait complete. 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.843+0000 7fe1dc285640 1 Processor -- start 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.843+0000 7fe1dc285640 1 -- start start 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1dc285640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1d41016c0 0x7fe1d41966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1dc285640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d4196c00 0x7fe1d419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1dc285640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d419c4e0 0x7fe1d419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1dc285640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d410ade0 con 0x7fe1d419c4e0 2024-09-16T08:54:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1dc285640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d410ac60 con 0x7fe1d41016c0 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1dc285640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d410af60 con 0x7fe1d4196c00 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da822640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d419c4e0 0x7fe1d419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da822640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d419c4e0 0x7fe1d419c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40482/0 (socket says 172.21.15.111:40482) 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da822640 1 -- 172.21.15.111:0/1496746914 learned_addr learned my addr 172.21.15.111:0/1496746914 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1d9820640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d4196c00 0x7fe1d419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da021640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1d41016c0 0x7fe1d41966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da822640 1 -- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d4196c00 msgr2=0x7fe1d419bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da822640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d4196c00 0x7fe1d419bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da822640 1 -- 172.21.15.111:0/1496746914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1d41016c0 msgr2=0x7fe1d41966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da822640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1d41016c0 0x7fe1d41966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da822640 1 -- 172.21.15.111:0/1496746914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d41a3120 con 0x7fe1d419c4e0 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da021640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1d41016c0 0x7fe1d41966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T08:54:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1da822640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d419c4e0 0x7fe1d419c990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00ba50 tx=0x7fe1cc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1c77fe640 1 -- 172.21.15.111:0/1496746914 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc019070 con 0x7fe1d419c4e0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1c77fe640 1 -- 172.21.15.111:0/1496746914 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fe1cc00cbe0 con 0x7fe1d419c4e0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d41a3410 con 0x7fe1d419c4e0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.844+0000 7fe1c77fe640 1 -- 172.21.15.111:0/1496746914 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc0075b0 con 0x7fe1d419c4e0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.845+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d41a38d0 con 0x7fe1d419c4e0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.845+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d4101b20 con 0x7fe1d419c4e0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.846+0000 7fe1c77fe640 1 -- 172.21.15.111:0/1496746914 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7fe1cc01c070 con 0x7fe1d419c4e0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.846+0000 7fe1c77fe640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe1c003fbd0 0x7fe1c0042090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.846+0000 7fe1c77fe640 1 -- 172.21.15.111:0/1496746914 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe1cc053810 con 0x7fe1d419c4e0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.846+0000 7fe1da021640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe1c003fbd0 0x7fe1c0042090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.846+0000 7fe1da021640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe1c003fbd0 0x7fe1c0042090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0000c00 tx=0x7fe1d002aad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:52.850+0000 7fe1c77fe640 1 -- 172.21.15.111:0/1496746914 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fe1cc007c70 con 0x7fe1d419c4e0 2024-09-16T08:54:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: Updating smithi111:/etc/ceph/ceph.conf 2024-09-16T08:54:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-16T08:54:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: Reconfiguring daemon mon.a on smithi111 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: Updating smithi111:/etc/ceph/ceph.conf 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-16T08:54:52.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:52.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:52.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T08:54:52.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: Reconfiguring daemon mon.a on smithi111 2024-09-16T08:54:52.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:52.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:52.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T08:54:52.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:53.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.037+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fe1d41a3b90 con 0x7fe1d419c4e0 2024-09-16T08:54:53.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.037+0000 7fe1c77fe640 1 -- 172.21.15.111:0/1496746914 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7fe1cc007750 con 0x7fe1d419c4e0 2024-09-16T08:54:53.040 INFO:teuthology.orchestra.run.smithi111.stdout:# minimal ceph.conf for f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:53.040 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2024-09-16T08:54:53.040 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T08:54:53.040 INFO:teuthology.orchestra.run.smithi111.stdout: mon_host = [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] 2024-09-16T08:54:53.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe1c003fbd0 msgr2=0x7fe1c0042090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:53.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe1c003fbd0 0x7fe1c0042090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0000c00 tx=0x7fe1d002aad0 comp rx=0 tx=0).stop 2024-09-16T08:54:53.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d419c4e0 msgr2=0x7fe1d419c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:53.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d419c4e0 0x7fe1d419c990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00ba50 tx=0x7fe1cc00bf10 comp rx=0 tx=0).stop 2024-09-16T08:54:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 shutdown_connections 2024-09-16T08:54:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe1c003fbd0 0x7fe1c0042090 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d419c4e0 0x7fe1d419c990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d4196c00 0x7fe1d419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 --2- 172.21.15.111:0/1496746914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1d41016c0 0x7fe1d41966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 >> 172.21.15.111:0/1496746914 conn(0x7fe1d40fd130 msgr2=0x7fe1d4102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 shutdown_connections 2024-09-16T08:54:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:54:53.040+0000 7fe1dc285640 1 -- 172.21.15.111:0/1496746914 wait complete. 2024-09-16T08:54:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: Updating smithi111:/etc/ceph/ceph.conf 2024-09-16T08:54:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-16T08:54:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:54:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T08:54:53.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: Reconfiguring daemon mon.a on smithi111 2024-09-16T08:54:53.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:53.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:53.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:53.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:53.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T08:54:53.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:53.919 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:53 smithi111 ceph-mon[29803]: Reconfiguring mon.c (monmap changed)... 2024-09-16T08:54:53.920 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:53 smithi111 ceph-mon[29803]: Reconfiguring daemon mon.c on smithi111 2024-09-16T08:54:53.920 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1496746914' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:53.920 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:53 smithi111 ceph-mon[40056]: Reconfiguring mon.c (monmap changed)... 2024-09-16T08:54:53.920 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:53 smithi111 ceph-mon[40056]: Reconfiguring daemon mon.c on smithi111 2024-09-16T08:54:53.920 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1496746914' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:54.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:53 smithi114 ceph-mon[33315]: Reconfiguring mon.c (monmap changed)... 2024-09-16T08:54:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:53 smithi114 ceph-mon[33315]: Reconfiguring daemon mon.c on smithi111 2024-09-16T08:54:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1496746914' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:54.200 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-16T08:54:54.200 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:54:54.200 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T08:54:54.229 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:54:54.229 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:54:54.295 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:54:54.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T08:54:54.329 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:54:54.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:54:54.399 INFO:tasks.cephadm:Adding mgr.y on smithi111 2024-09-16T08:54:54.399 INFO:tasks.cephadm:Adding mgr.x on smithi114 2024-09-16T08:54:54.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi111=y;smithi114=x' 2024-09-16T08:54:54.683 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:54:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[29803]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T08:54:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:54.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[40056]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:54.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:54.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:54.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:54.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T08:54:54.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:54 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:55.051 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:54 smithi114 ceph-mon[33315]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:55.051 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:54 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:55.051 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:54 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:55.052 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:54 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-16T08:54:55.052 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:54 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-16T08:54:55.052 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:54 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:55.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.133+0000 7f03edb2d640 1 -- 172.21.15.114:0/1323325032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e806bad0 msgr2=0x7f03e81051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:55.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 --2- 172.21.15.114:0/1323325032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e806bad0 0x7f03e81051b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009880 tx=0x7f03d802a5d0 comp rx=0 tx=0).stop 2024-09-16T08:54:55.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 -- 172.21.15.114:0/1323325032 shutdown_connections 2024-09-16T08:54:55.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 --2- 172.21.15.114:0/1323325032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e806bad0 0x7f03e81051b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:55.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 -- 172.21.15.114:0/1323325032 >> 172.21.15.114:0/1323325032 conn(0x7f03e80fcd00 msgr2=0x7f03e80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:55.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 -- 172.21.15.114:0/1323325032 shutdown_connections 2024-09-16T08:54:55.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 -- 172.21.15.114:0/1323325032 wait complete. 2024-09-16T08:54:55.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 Processor -- start 2024-09-16T08:54:55.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 -- start start 2024-09-16T08:54:55.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03e806bad0 0x7f03e8197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:55.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e8197910 0x7f03e81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:55.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e8198b30 0x7f03e8198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f03e8107990 con 0x7f03e8197910 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f03e8107810 con 0x7f03e8198b30 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.134+0000 7f03edb2d640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f03e8107b10 con 0x7f03e806bad0 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e8197910 0x7f03e81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e77fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03e806bad0 0x7f03e8197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e8197910 0x7f03e81985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.114:58644/0 (socket says 172.21.15.114:58644) 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e6ffd640 1 -- 172.21.15.114:0/1946257321 learned_addr learned my addr 172.21.15.114:0/1946257321 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e7fff640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e8198b30 0x7f03e8198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e6ffd640 1 -- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03e806bad0 msgr2=0x7f03e8197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e6ffd640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03e806bad0 0x7f03e8197360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e6ffd640 1 -- 172.21.15.114:0/1946257321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e8198b30 msgr2=0x7f03e8198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e6ffd640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e8198b30 0x7f03e8198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e6ffd640 1 -- 172.21.15.114:0/1946257321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e819f710 con 0x7f03e8197910 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e6ffd640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e8197910 0x7f03e81985f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00b550 tx=0x7f03dc00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e4ff9640 1 -- 172.21.15.114:0/1946257321 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc004230 con 0x7f03e8197910 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e4ff9640 1 -- 172.21.15.114:0/1946257321 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f03dc0043d0 con 0x7f03e8197910 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e819fa00 con 0x7f03e8197910 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03e4ff9640 1 -- 172.21.15.114:0/1946257321 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0039b0 con 0x7f03e8197910 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.135+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e819ffc0 con 0x7f03e8197910 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.136+0000 7f03e4ff9640 1 -- 172.21.15.114:0/1946257321 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7f03dc003b50 con 0x7f03e8197910 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.136+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e8103220 con 0x7f03e8197910 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.136+0000 7f03e4ff9640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f03d003fbd0 0x7f03d0042090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.137+0000 7f03e4ff9640 1 -- 172.21.15.114:0/1946257321 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f03dc053ac0 con 0x7f03e8197910 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.137+0000 7f03e77fe640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f03d003fbd0 0x7f03d0042090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:54:55.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.137+0000 7f03e77fe640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f03d003fbd0 0x7f03d0042090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f03d8000c00 tx=0x7f03d80095a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:54:55.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.141+0000 7f03e4ff9640 1 -- 172.21.15.114:0/1946257321 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f03dc013070 con 0x7f03e8197910 2024-09-16T08:54:55.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.336+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi111=y;smithi114=x", "target": ["mon-mgr", ""]}) -- 0x7f03e8063590 con 0x7f03d003fbd0 2024-09-16T08:54:55.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.342+0000 7f03e4ff9640 1 -- 172.21.15.114:0/1946257321 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f03e8063590 con 0x7f03d003fbd0 2024-09-16T08:54:55.343 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled mgr update... 2024-09-16T08:54:55.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.344+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f03d003fbd0 msgr2=0x7f03d0042090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:55.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.344+0000 7f03edb2d640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f03d003fbd0 0x7f03d0042090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f03d8000c00 tx=0x7f03d80095a0 comp rx=0 tx=0).stop 2024-09-16T08:54:55.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.344+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e8197910 msgr2=0x7f03e81985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:54:55.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.344+0000 7f03edb2d640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e8197910 0x7f03e81985f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00b550 tx=0x7f03dc00ba10 comp rx=0 tx=0).stop 2024-09-16T08:54:55.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.345+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 shutdown_connections 2024-09-16T08:54:55.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.345+0000 7f03edb2d640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f03d003fbd0 0x7f03d0042090 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:55.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.345+0000 7f03edb2d640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03e8198b30 0x7f03e8198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:55.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.345+0000 7f03edb2d640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03e8197910 0x7f03e81985f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:55.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.345+0000 7f03edb2d640 1 --2- 172.21.15.114:0/1946257321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03e806bad0 0x7f03e8197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:54:55.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.345+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 >> 172.21.15.114:0/1946257321 conn(0x7f03e80fcd00 msgr2=0x7f03e80fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:54:55.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.345+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 shutdown_connections 2024-09-16T08:54:55.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:54:55.345+0000 7f03edb2d640 1 -- 172.21.15.114:0/1946257321 wait complete. 2024-09-16T08:54:56.074 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:55 smithi114 ceph-mon[33315]: Reconfiguring mon.b (monmap changed)... 2024-09-16T08:54:56.074 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:55 smithi114 ceph-mon[33315]: Reconfiguring daemon mon.b on smithi114 2024-09-16T08:54:56.074 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:55 smithi111 ceph-mon[29803]: Reconfiguring mon.b (monmap changed)... 2024-09-16T08:54:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:55 smithi111 ceph-mon[29803]: Reconfiguring daemon mon.b on smithi114 2024-09-16T08:54:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:56.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:55 smithi111 ceph-mon[40056]: Reconfiguring mon.b (monmap changed)... 2024-09-16T08:54:56.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:55 smithi111 ceph-mon[40056]: Reconfiguring daemon mon.b on smithi114 2024-09-16T08:54:56.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:56.359 DEBUG:teuthology.orchestra.run.smithi114:mgr.x> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.x.service 2024-09-16T08:54:56.362 DEBUG:tasks.cephadm:set 0 configs 2024-09-16T08:54:56.362 INFO:tasks.cephadm:Deploying OSDs... 2024-09-16T08:54:56.362 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:54:56.362 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T08:54:56.382 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T08:54:56.383 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-09-16T08:54:56.441 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T08:54:56.442 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T08:54:56.442 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 960 Links: 1 2024-09-16T08:54:56.442 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T08:54:56.442 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T08:54:56.442 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-09-16 08:54:44.060938423 +0000 2024-09-16T08:54:56.442 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-09-16 08:49:54.806261991 +0000 2024-09-16T08:54:56.442 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-09-16 08:49:54.806261991 +0000 2024-09-16T08:54:56.442 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-09-16 08:49:54.806261991 +0000 2024-09-16T08:54:56.443 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T08:54:56.511 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-09-16T08:54:56.511 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-09-16T08:54:56.511 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000237324 s, 2.2 MB/s 2024-09-16T08:54:56.513 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T08:54:56.572 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-09-16T08:54:56.630 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T08:54:56.630 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T08:54:56.630 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-16T08:54:56.630 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T08:54:56.630 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T08:54:56.631 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-09-16 08:54:44.061938405 +0000 2024-09-16T08:54:56.631 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-09-16 08:49:54.808261953 +0000 2024-09-16T08:54:56.631 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-09-16 08:49:54.808261953 +0000 2024-09-16T08:54:56.631 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-09-16 08:49:54.808261953 +0000 2024-09-16T08:54:56.631 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T08:54:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-09-16T08:54:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-09-16T08:54:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000225769 s, 2.3 MB/s 2024-09-16T08:54:56.699 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T08:54:56.757 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-09-16T08:54:56.814 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T08:54:56.814 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T08:54:56.814 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 965 Links: 1 2024-09-16T08:54:56.814 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T08:54:56.814 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T08:54:56.814 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-09-16 08:54:44.061938405 +0000 2024-09-16T08:54:56.815 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-09-16 08:49:54.807261972 +0000 2024-09-16T08:54:56.815 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-09-16 08:49:54.807261972 +0000 2024-09-16T08:54:56.815 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-09-16 08:49:54.807261972 +0000 2024-09-16T08:54:56.815 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T08:54:56.882 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-09-16T08:54:56.882 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-09-16T08:54:56.882 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.00238658 s, 215 kB/s 2024-09-16T08:54:56.883 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T08:54:56.941 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-09-16T08:54:56.996 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T08:54:56.997 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T08:54:56.997 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-16T08:54:56.997 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T08:54:56.997 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T08:54:56.997 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-09-16 08:54:44.062938387 +0000 2024-09-16T08:54:56.997 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-09-16 08:49:54.808261953 +0000 2024-09-16T08:54:56.997 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-09-16 08:49:54.808261953 +0000 2024-09-16T08:54:56.997 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-09-16 08:49:54.808261953 +0000 2024-09-16T08:54:56.997 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T08:54:57.058 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:57.058 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi111=y;smithi114=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: Saving service mgr spec with placement smithi111=y;smithi114=x;count:2 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T08:54:57.059 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi111=y;smithi114=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: Saving service mgr spec with placement smithi111=y;smithi114=x;count:2 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T08:54:57.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T08:54:57.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T08:54:57.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:57.062 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-09-16T08:54:57.062 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-09-16T08:54:57.062 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000176638 s, 2.9 MB/s 2024-09-16T08:54:57.062 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T08:54:57.119 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:54:57.119 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T08:54:57.136 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T08:54:57.136 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-09-16T08:54:57.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:57.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi111=y;smithi114=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:54:57.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: Saving service mgr spec with placement smithi111=y;smithi114=x;count:2 2024-09-16T08:54:57.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:57.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:57.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:54:57.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:57.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:54:57.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:54:57.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T08:54:57.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T08:54:57.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T08:54:57.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:54:57.170 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T08:54:57.170 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T08:54:57.170 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-16T08:54:57.170 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T08:54:57.170 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T08:54:57.170 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 08:54:35.834789831 +0000 2024-09-16T08:54:57.170 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 08:49:53.267238439 +0000 2024-09-16T08:54:57.170 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 08:49:53.267238439 +0000 2024-09-16T08:54:57.171 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-16 08:49:53.267238439 +0000 2024-09-16T08:54:57.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T08:54:57.234 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T08:54:57.234 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T08:54:57.234 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000284636 s, 1.8 MB/s 2024-09-16T08:54:57.236 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T08:54:57.293 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-09-16T08:54:57.349 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T08:54:57.349 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T08:54:57.349 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 968 Links: 1 2024-09-16T08:54:57.349 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T08:54:57.349 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T08:54:57.350 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 08:54:35.834789831 +0000 2024-09-16T08:54:57.350 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 08:49:53.267238439 +0000 2024-09-16T08:54:57.350 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 08:49:53.267238439 +0000 2024-09-16T08:54:57.350 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-16 08:49:53.267238439 +0000 2024-09-16T08:54:57.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T08:54:57.413 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T08:54:57.414 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T08:54:57.414 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000204884 s, 2.5 MB/s 2024-09-16T08:54:57.415 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T08:54:57.472 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-09-16T08:54:57.528 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T08:54:57.528 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T08:54:57.528 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 977 Links: 1 2024-09-16T08:54:57.529 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T08:54:57.529 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T08:54:57.529 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 08:54:35.834789831 +0000 2024-09-16T08:54:57.529 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 08:49:53.267238439 +0000 2024-09-16T08:54:57.529 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 08:49:53.267238439 +0000 2024-09-16T08:54:57.529 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-16 08:49:53.267238439 +0000 2024-09-16T08:54:57.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T08:54:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T08:54:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T08:54:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000175095 s, 2.9 MB/s 2024-09-16T08:54:57.594 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T08:54:57.650 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-09-16T08:54:57.706 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T08:54:57.706 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T08:54:57.706 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 983 Links: 1 2024-09-16T08:54:57.706 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T08:54:57.706 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T08:54:57.706 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-16 08:54:35.835789808 +0000 2024-09-16T08:54:57.706 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-16 08:49:53.268238166 +0000 2024-09-16T08:54:57.707 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-16 08:49:53.268238166 +0000 2024-09-16T08:54:57.707 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-16 08:49:53.268238166 +0000 2024-09-16T08:54:57.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T08:54:57.769 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-16T08:54:57.770 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-16T08:54:57.770 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000229929 s, 2.2 MB/s 2024-09-16T08:54:57.771 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T08:54:57.827 INFO:tasks.cephadm:Deploying osd.0 on smithi111 with /dev/vg_nvme/lv_4... 2024-09-16T08:54:57.828 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T08:54:58.071 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:54:58.125 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:57 smithi114 ceph-mon[33315]: Deploying daemon mgr.x on smithi114 2024-09-16T08:54:58.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:57 smithi111 ceph-mon[29803]: Deploying daemon mgr.x on smithi114 2024-09-16T08:54:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:57 smithi111 ceph-mon[40056]: Deploying daemon mgr.x on smithi114 2024-09-16T08:54:58.663 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:54:58 smithi114 systemd[1]: Starting Ceph mgr.x for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T08:54:59.071 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:54:58 smithi114 ceph-mon[33315]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:59.072 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:54:58 smithi114 podman[34784]: 2024-09-16 08:54:58.76089513 +0000 UTC m=+0.111590432 container create f108ac6811d146ca0a8c3bc199c0e68d6dbeedd2bfe2806af34558ad7b6e05cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-16T08:54:59.072 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:54:58 smithi114 podman[34784]: 2024-09-16 08:54:58.664989497 +0000 UTC m=+0.015684764 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T08:54:59.072 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:54:58 smithi114 podman[34784]: 2024-09-16 08:54:58.963241074 +0000 UTC m=+0.313936334 container init f108ac6811d146ca0a8c3bc199c0e68d6dbeedd2bfe2806af34558ad7b6e05cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=) 2024-09-16T08:54:59.072 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:54:58 smithi114 podman[34784]: 2024-09-16 08:54:58.968356554 +0000 UTC m=+0.319051815 container start f108ac6811d146ca0a8c3bc199c0e68d6dbeedd2bfe2806af34558ad7b6e05cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-16T08:54:59.072 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:54:58 smithi114 ceph-mgr[34822]: -- 172.21.15.114:0/4236410344 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556a048f50e0 con 0x556a048d3400 2024-09-16T08:54:59.072 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:54:59 smithi114 bash[34784]: f108ac6811d146ca0a8c3bc199c0e68d6dbeedd2bfe2806af34558ad7b6e05cd 2024-09-16T08:54:59.119 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:54:58 smithi111 ceph-mon[29803]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:59.119 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:54:58 smithi111 ceph-mon[40056]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:54:59.413 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:54:59 smithi114 systemd[1]: Started Ceph mgr.x for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:55:00.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:00 smithi114 ceph-mon[33315]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:00.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:00 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:00 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:00 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:00 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:00 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[29803]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:00.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[40056]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:00.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:00.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:00 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:00.738 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:55:00.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2024-09-16T08:55:01.513 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:55:01.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:01.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: Reconfiguring mgr.y (unknown last config time)... 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[29803]: Reconfiguring daemon mgr.y on smithi111 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:01.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:01.609 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:01.609 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:01.609 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: Reconfiguring mgr.y (unknown last config time)... 2024-09-16T08:55:01.609 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T08:55:01.609 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T08:55:01.609 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:01.609 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:01 smithi111 ceph-mon[40056]: Reconfiguring daemon mgr.y on smithi111 2024-09-16T08:55:01.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: Reconfiguring mgr.y (unknown last config time)... 2024-09-16T08:55:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-16T08:55:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-16T08:55:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:01 smithi114 ceph-mon[33315]: Reconfiguring daemon mgr.y on smithi111 2024-09-16T08:55:02.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.404+0000 7f960216b640 1 -- 172.21.15.111:0/54565038 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc104520 msgr2=0x7f95fc10aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:02.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.404+0000 7f960216b640 1 --2- 172.21.15.111:0/54565038 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc104520 0x7f95fc10aea0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009c40 tx=0x7f95ec02a710 comp rx=0 tx=0).stop 2024-09-16T08:55:02.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.404+0000 7f960216b640 1 -- 172.21.15.111:0/54565038 shutdown_connections 2024-09-16T08:55:02.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.404+0000 7f960216b640 1 --2- 172.21.15.111:0/54565038 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc104520 0x7f95fc10aea0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:02.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.404+0000 7f960216b640 1 --2- 172.21.15.111:0/54565038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95fc103b60 0x7f95fc103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.404+0000 7f960216b640 1 --2- 172.21.15.111:0/54565038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc102a00 0x7f95fc102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.404+0000 7f960216b640 1 -- 172.21.15.111:0/54565038 >> 172.21.15.111:0/54565038 conn(0x7f95fc0fe2f0 msgr2=0x7f95fc100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.404+0000 7f960216b640 1 -- 172.21.15.111:0/54565038 shutdown_connections 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.404+0000 7f960216b640 1 -- 172.21.15.111:0/54565038 wait complete. 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f960216b640 1 Processor -- start 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f960216b640 1 -- start start 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f960216b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc102a00 0x7f95fc1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f960216b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95fc103b60 0x7f95fc198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f960216b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc104520 0x7f95fc199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f960216b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc10daf0 con 0x7f95fc103b60 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f960216b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc10d970 con 0x7f95fc104520 2024-09-16T08:55:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f960216b640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f95fc10dc70 con 0x7f95fc102a00 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fb7fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc102a00 0x7f95fc1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fb7fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc102a00 0x7f95fc1983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50916/0 (socket says 172.21.15.111:50916) 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95faffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95fc103b60 0x7f95fc198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fb7fe640 1 -- 172.21.15.111:0/2878804104 learned_addr learned my addr 172.21.15.111:0/2878804104 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fbfff640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc104520 0x7f95fc199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fb7fe640 1 -- 172.21.15.111:0/2878804104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc104520 msgr2=0x7f95fc199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fb7fe640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc104520 0x7f95fc199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fb7fe640 1 -- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95fc103b60 msgr2=0x7f95fc198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fb7fe640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95fc103b60 0x7f95fc198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fb7fe640 1 -- 172.21.15.111:0/2878804104 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95fc199ee0 con 0x7f95fc102a00 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.405+0000 7f95fb7fe640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc102a00 0x7f95fc1983b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f95f400e9f0 tx=0x7f95f400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.406+0000 7f95f8ff9640 1 -- 172.21.15.111:0/2878804104 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f400cd80 con 0x7f95fc102a00 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.406+0000 7f95f8ff9640 1 -- 172.21.15.111:0/2878804104 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f95f4005470 con 0x7f95fc102a00 2024-09-16T08:55:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.406+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95fc19a1d0 con 0x7f95fc102a00 2024-09-16T08:55:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.406+0000 7f95f8ff9640 1 -- 172.21.15.111:0/2878804104 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4010430 con 0x7f95fc102a00 2024-09-16T08:55:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.406+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f95fc1a08b0 con 0x7f95fc102a00 2024-09-16T08:55:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.407+0000 7f95f8ff9640 1 -- 172.21.15.111:0/2878804104 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 13) ==== 51822+0+0 (secure 0 0 0) 0x7f95f40105d0 con 0x7f95fc102a00 2024-09-16T08:55:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.407+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95fc102e00 con 0x7f95fc102a00 2024-09-16T08:55:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.407+0000 7f95f8ff9640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f95e403f980 0x7f95e4041e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:55:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.407+0000 7f95f8ff9640 1 -- 172.21.15.111:0/2878804104 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f95f40547b0 con 0x7f95fc102a00 2024-09-16T08:55:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.407+0000 7f95faffd640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f95e403f980 0x7f95e4041e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:55:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.407+0000 7f95faffd640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f95e403f980 0x7f95e4041e40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f95f00096c0 tx=0x7f95f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:55:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.412+0000 7f95f8ff9640 1 -- 172.21.15.111:0/2878804104 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f95f4016330 con 0x7f95fc102a00 2024-09-16T08:55:02.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:02.597+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f95fc063590 con 0x7f95e403f980 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[29803]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[40056]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:02.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:02.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:02.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:02.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:02 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:02 smithi114 ceph-mon[33315]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:02 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:02 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:02 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:02 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:02 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:02 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:02 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:03 smithi114 ceph-mon[33315]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:55:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:03 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:55:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:03 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:55:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:03 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:03 smithi111 ceph-mon[29803]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:55:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:03 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:55:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:03 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:55:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:03 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:03 smithi111 ceph-mon[40056]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:55:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:03 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:55:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:03 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:55:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:03 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:04.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:04.286+0000 7f95f8ff9640 1 -- 172.21.15.111:0/2878804104 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mgrmap(e 14) ==== 103024+0+0 (secure 0 0 0) 0x7f95f40199c0 con 0x7f95fc102a00 2024-09-16T08:55:04.660 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[29803]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[29803]: Standby manager daemon x started 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[29803]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[29803]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[29803]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[29803]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[40056]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[40056]: Standby manager daemon x started 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[40056]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[40056]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T08:55:04.661 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[40056]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T08:55:04.662 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:04 smithi111 ceph-mon[40056]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T08:55:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:04 smithi114 ceph-mon[33315]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:04 smithi114 ceph-mon[33315]: Standby manager daemon x started 2024-09-16T08:55:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:04 smithi114 ceph-mon[33315]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T08:55:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:04 smithi114 ceph-mon[33315]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T08:55:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:04 smithi114 ceph-mon[33315]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T08:55:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:04 smithi114 ceph-mon[33315]: from='mgr.? 172.21.15.114:0/3568421154' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T08:55:05.539 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[29803]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-16T08:55:05.539 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T08:55:05.539 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/654798310' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b1e3493f-2bf8-489d-b890-6cba7976e3c7"} : dispatch 2024-09-16T08:55:05.540 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/654798310' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1e3493f-2bf8-489d-b890-6cba7976e3c7"}]': finished 2024-09-16T08:55:05.540 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[29803]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T08:55:05.540 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:05.540 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[40056]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-16T08:55:05.540 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T08:55:05.540 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/654798310' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b1e3493f-2bf8-489d-b890-6cba7976e3c7"} : dispatch 2024-09-16T08:55:05.540 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/654798310' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1e3493f-2bf8-489d-b890-6cba7976e3c7"}]': finished 2024-09-16T08:55:05.540 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[40056]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T08:55:05.540 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:05 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:05.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:05 smithi114 ceph-mon[33315]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-16T08:55:05.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:05 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T08:55:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/654798310' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b1e3493f-2bf8-489d-b890-6cba7976e3c7"} : dispatch 2024-09-16T08:55:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/654798310' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1e3493f-2bf8-489d-b890-6cba7976e3c7"}]': finished 2024-09-16T08:55:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:05 smithi114 ceph-mon[33315]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T08:55:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:05 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:06 smithi114 ceph-mon[33315]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3702209895' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:55:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:06 smithi111 ceph-mon[29803]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3702209895' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:55:06.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:06 smithi111 ceph-mon[40056]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:06.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3702209895' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:55:08.609 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:08 smithi111 ceph-mon[29803]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:08.610 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:08 smithi111 ceph-mon[40056]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:08.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:08 smithi114 ceph-mon[33315]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:10.566 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:10 smithi111 ceph-mon[29803]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:10.566 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:10 smithi111 ceph-mon[40056]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:10.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:10 smithi114 ceph-mon[33315]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:12.483 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:12 smithi111 ceph-mon[40056]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:12.484 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:12 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-16T08:55:12.484 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:12 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:12.484 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:12 smithi111 ceph-mon[40056]: Deploying daemon osd.0 on smithi111 2024-09-16T08:55:12.484 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:12 smithi111 ceph-mon[29803]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:12.484 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:12 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-16T08:55:12.484 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:12 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:12.484 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:12 smithi111 ceph-mon[29803]: Deploying daemon osd.0 on smithi111 2024-09-16T08:55:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:12 smithi114 ceph-mon[33315]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:12 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-16T08:55:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:12 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:12 smithi114 ceph-mon[33315]: Deploying daemon osd.0 on smithi111 2024-09-16T08:55:14.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:14 smithi111 ceph-mon[29803]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:14.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:14 smithi111 ceph-mon[40056]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:14.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:14 smithi114 ceph-mon[33315]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:16.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:16 smithi114 ceph-mon[33315]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:16 smithi111 ceph-mon[29803]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:16 smithi111 ceph-mon[40056]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:18.526 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:18 smithi111 ceph-mon[29803]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:18.526 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:18 smithi111 ceph-mon[40056]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:18.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:18 smithi114 ceph-mon[33315]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:20.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:20 smithi114 ceph-mon[33315]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:20.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:20 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:20 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:20 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:20 smithi111 ceph-mon[29803]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:20 smithi111 ceph-mon[40056]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:20 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:20 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:20 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:22.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:22 smithi114 ceph-mon[33315]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:22 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:22 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:22.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:22 smithi111 ceph-mon[29803]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:22.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:22 smithi111 ceph-mon[40056]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:22.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:22 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:22.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:22 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:23.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:23 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:23 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:23 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:23 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:23 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:23 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:23 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:23 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:23 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:24.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.691+0000 7f95f8ff9640 1 -- 172.21.15.111:0/2878804104 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f95fc063590 con 0x7f95e403f980 2024-09-16T08:55:24.693 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 0 on host 'smithi111' 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f95e403f980 msgr2=0x7f95e4041e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f95e403f980 0x7f95e4041e40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f95f00096c0 tx=0x7f95f0009260 comp rx=0 tx=0).stop 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc102a00 msgr2=0x7f95fc1983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc102a00 0x7f95fc1983b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f95f400e9f0 tx=0x7f95f400edf0 comp rx=0 tx=0).stop 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 shutdown_connections 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f95e403f980 0x7f95e4041e40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc104520 0x7f95fc199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95fc103b60 0x7f95fc198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 --2- 172.21.15.111:0/2878804104 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95fc102a00 0x7f95fc1983b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 >> 172.21.15.111:0/2878804104 conn(0x7f95fc0fe2f0 msgr2=0x7f95fc0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:55:24.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 shutdown_connections 2024-09-16T08:55:24.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:24.695+0000 7f960216b640 1 -- 172.21.15.111:0/2878804104 wait complete. 2024-09-16T08:55:24.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:24 smithi114 ceph-mon[33315]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:24.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:24 smithi114 ceph-mon[33315]: from='osd.0 [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T08:55:24.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:24 smithi114 ceph-mon[33315]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T08:55:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:24 smithi111 ceph-mon[29803]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:24 smithi111 ceph-mon[29803]: from='osd.0 [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T08:55:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:24 smithi111 ceph-mon[29803]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T08:55:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:24 smithi111 ceph-mon[40056]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:24 smithi111 ceph-mon[40056]: from='osd.0 [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T08:55:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:24 smithi111 ceph-mon[40056]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-16T08:55:25.370 DEBUG:teuthology.orchestra.run.smithi111:osd.0> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.0.service 2024-09-16T08:55:25.373 INFO:tasks.cephadm:Deploying osd.1 on smithi111 with /dev/vg_nvme/lv_3... 2024-09-16T08:55:25.373 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T08:55:25.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:25 smithi114 ceph-mon[33315]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T08:55:25.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:25 smithi114 ceph-mon[33315]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T08:55:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:25 smithi114 ceph-mon[33315]: from='osd.0 [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:55:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:25 smithi114 ceph-mon[33315]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:55:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:25 smithi114 ceph-mon[33315]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:55:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[29803]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T08:55:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[29803]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T08:55:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[29803]: from='osd.0 [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:55:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[29803]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:55:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[29803]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[40056]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[40056]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[40056]: from='osd.0 [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[40056]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-mon[40056]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:55:25.928 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0[43720]: 2024-09-16T08:55:25.642+0000 7fd992635640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T08:55:25.928 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:55:25 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0[43720]: 2024-09-16T08:55:25.646+0000 7fd98e45c640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T08:55:26.161 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:55:26.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:26 smithi114 ceph-mon[33315]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:26.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:26 smithi114 ceph-mon[33315]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T08:55:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:26 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:26 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:26.925 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:26 smithi111 ceph-mon[29803]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:26 smithi111 ceph-mon[29803]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T08:55:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:26 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:26 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:26 smithi111 ceph-mon[40056]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:26 smithi111 ceph-mon[40056]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T08:55:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:26 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:26 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[29803]: purged_snaps scrub starts 2024-09-16T08:55:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[29803]: purged_snaps scrub ok 2024-09-16T08:55:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[29803]: osd.0 [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] boot 2024-09-16T08:55:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[29803]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T08:55:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[40056]: purged_snaps scrub starts 2024-09-16T08:55:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[40056]: purged_snaps scrub ok 2024-09-16T08:55:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[40056]: osd.0 [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] boot 2024-09-16T08:55:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[40056]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T08:55:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:27.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:27 smithi114 ceph-mon[33315]: purged_snaps scrub starts 2024-09-16T08:55:27.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:27 smithi114 ceph-mon[33315]: purged_snaps scrub ok 2024-09-16T08:55:27.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:27.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:27 smithi114 ceph-mon[33315]: osd.0 [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] boot 2024-09-16T08:55:27.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:27 smithi114 ceph-mon[33315]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T08:55:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:55:28.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:28 smithi114 ceph-mon[33315]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:28 smithi114 ceph-mon[33315]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T08:55:28.925 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:28 smithi111 ceph-mon[29803]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:28.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:28 smithi111 ceph-mon[29803]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T08:55:28.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:28 smithi111 ceph-mon[40056]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T08:55:28.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:28 smithi111 ceph-mon[40056]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T08:55:30.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:30.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: Detected new or changed devices on smithi111 2024-09-16T08:55:30.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi111 to 16064M 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: Detected new or changed devices on smithi111 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi111 to 16064M 2024-09-16T08:55:30.616 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.617 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:30.617 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:30.617 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:30.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: Detected new or changed devices on smithi111 2024-09-16T08:55:30.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:55:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi111 to 16064M 2024-09-16T08:55:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:30.940 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:55:30.966 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2024-09-16T08:55:31.221 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:55:31.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.720+0000 7f52fcdad640 1 -- 172.21.15.111:0/4051502446 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52f8104520 msgr2=0x7f52f810aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.720+0000 7f52fcdad640 1 --2- 172.21.15.111:0/4051502446 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52f8104520 0x7f52f810aea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009d00 tx=0x7f52f002a910 comp rx=0 tx=0).stop 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.720+0000 7f52fcdad640 1 -- 172.21.15.111:0/4051502446 shutdown_connections 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.720+0000 7f52fcdad640 1 --2- 172.21.15.111:0/4051502446 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52f8104520 0x7f52f810aea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.720+0000 7f52fcdad640 1 --2- 172.21.15.111:0/4051502446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8103b60 0x7f52f8103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.720+0000 7f52fcdad640 1 --2- 172.21.15.111:0/4051502446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a00 0x7f52f8102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.720+0000 7f52fcdad640 1 -- 172.21.15.111:0/4051502446 >> 172.21.15.111:0/4051502446 conn(0x7f52f80fe2f0 msgr2=0x7f52f8100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.720+0000 7f52fcdad640 1 -- 172.21.15.111:0/4051502446 shutdown_connections 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.720+0000 7f52fcdad640 1 -- 172.21.15.111:0/4051502446 wait complete. 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.721+0000 7f52fcdad640 1 Processor -- start 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.721+0000 7f52fcdad640 1 -- start start 2024-09-16T08:55:31.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.721+0000 7f52fcdad640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a00 0x7f52f8198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.721+0000 7f52fcdad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8103b60 0x7f52f8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.721+0000 7f52fcdad640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52f8104520 0x7f52f8199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.721+0000 7f52fcdad640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52f810db40 con 0x7f52f8102a00 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.721+0000 7f52fcdad640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f52f810d9c0 con 0x7f52f8103b60 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.721+0000 7f52fcdad640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f52f810dcc0 con 0x7f52f8104520 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.721+0000 7f52f659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a00 0x7f52f8198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f6d9d640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52f8104520 0x7f52f8199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a00 0x7f52f8198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43668/0 (socket says 172.21.15.111:43668) 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f659c640 1 -- 172.21.15.111:0/2231133796 learned_addr learned my addr 172.21.15.111:0/2231133796 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f5d9b640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8103b60 0x7f52f8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f659c640 1 -- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52f8104520 msgr2=0x7f52f8199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:31.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f659c640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52f8104520 0x7f52f8199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f659c640 1 -- 172.21.15.111:0/2231133796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8103b60 msgr2=0x7f52f8198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f659c640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8103b60 0x7f52f8198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f659c640 1 -- 172.21.15.111:0/2231133796 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f8199e30 con 0x7f52f8102a00 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52f659c640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a00 0x7f52f8198300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f52e800ea60 tx=0x7f52e800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52e77fe640 1 -- 172.21.15.111:0/2231133796 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e800cde0 con 0x7f52f8102a00 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f819a120 con 0x7f52f8102a00 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52e77fe640 1 -- 172.21.15.111:0/2231133796 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f52e8004470 con 0x7f52f8102a00 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52e77fe640 1 -- 172.21.15.111:0/2231133796 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8010430 con 0x7f52f8102a00 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.722+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f81a0800 con 0x7f52f8102a00 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.723+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f8102e00 con 0x7f52f8102a00 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.724+0000 7f52e77fe640 1 -- 172.21.15.111:0/2231133796 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 14) ==== 103024+0+0 (secure 0 0 0) 0x7f52e8003f60 con 0x7f52f8102a00 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.724+0000 7f52e77fe640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f52e007ac90 0x7f52e007d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:55:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.724+0000 7f52f5d9b640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f52e007ac90 0x7f52e007d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:55:31.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.724+0000 7f52e77fe640 1 -- 172.21.15.111:0/2231133796 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f52e809ce40 con 0x7f52f8102a00 2024-09-16T08:55:31.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.725+0000 7f52f5d9b640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f52e007ac90 0x7f52e007d150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f52ec005fd0 tx=0x7f52ec00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:55:31.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.729+0000 7f52e77fe640 1 -- 172.21.15.111:0/2231133796 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f52e80697e0 con 0x7f52f8102a00 2024-09-16T08:55:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:31.913+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f52f8063590 con 0x7f52e007ac90 2024-09-16T08:55:32.542 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:32 smithi111 ceph-mon[40056]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:32.543 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:32 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:55:32.543 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:32 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:55:32.543 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:32 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:32.543 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:32 smithi111 ceph-mon[29803]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:32.543 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:32 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:55:32.543 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:32 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:55:32.543 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:32 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:32.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:32 smithi114 ceph-mon[33315]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:32.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:32 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:55:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:32 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:55:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:32 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:33.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:33 smithi111 ceph-mon[29803]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:55:33.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:33 smithi111 ceph-mon[40056]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:55:33.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:33 smithi114 ceph-mon[33315]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:55:34.671 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:34 smithi114 ceph-mon[33315]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:34.671 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/710677232' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5c855195-4c6d-47c7-9c63-1e1cc3712c1e"} : dispatch 2024-09-16T08:55:34.671 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5c855195-4c6d-47c7-9c63-1e1cc3712c1e"} : dispatch 2024-09-16T08:55:34.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c855195-4c6d-47c7-9c63-1e1cc3712c1e"}]': finished 2024-09-16T08:55:34.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:34 smithi114 ceph-mon[33315]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T08:55:34.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:34 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[40056]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/710677232' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5c855195-4c6d-47c7-9c63-1e1cc3712c1e"} : dispatch 2024-09-16T08:55:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5c855195-4c6d-47c7-9c63-1e1cc3712c1e"} : dispatch 2024-09-16T08:55:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c855195-4c6d-47c7-9c63-1e1cc3712c1e"}]': finished 2024-09-16T08:55:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[40056]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T08:55:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:34.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[29803]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:34.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/710677232' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5c855195-4c6d-47c7-9c63-1e1cc3712c1e"} : dispatch 2024-09-16T08:55:34.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5c855195-4c6d-47c7-9c63-1e1cc3712c1e"} : dispatch 2024-09-16T08:55:34.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c855195-4c6d-47c7-9c63-1e1cc3712c1e"}]': finished 2024-09-16T08:55:34.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[29803]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T08:55:34.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:34 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:35.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/904804306' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:55:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/904804306' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:55:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/904804306' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:55:36.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:36 smithi114 ceph-mon[33315]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:36.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:36 smithi111 ceph-mon[29803]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:36.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:36 smithi111 ceph-mon[40056]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:38.404 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:38 smithi111 ceph-mon[29803]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:38.404 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:38 smithi111 ceph-mon[40056]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:38.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:38 smithi114 ceph-mon[33315]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:40.488 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:40 smithi111 ceph-mon[29803]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:40.488 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:40 smithi111 ceph-mon[40056]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:40.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:40 smithi114 ceph-mon[33315]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:41.657 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-16T08:55:41.658 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:41 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:41.658 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:41 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-16T08:55:41.658 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:41 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:41.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-16T08:55:41.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:41 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:42.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:42 smithi111 ceph-mon[29803]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:42.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:42 smithi111 ceph-mon[29803]: Deploying daemon osd.1 on smithi111 2024-09-16T08:55:42.615 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:42 smithi111 ceph-mon[40056]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:42.615 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:42 smithi111 ceph-mon[40056]: Deploying daemon osd.1 on smithi111 2024-09-16T08:55:42.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:42 smithi114 ceph-mon[33315]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:42.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:42 smithi114 ceph-mon[33315]: Deploying daemon osd.1 on smithi111 2024-09-16T08:55:44.643 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:44 smithi111 ceph-mon[29803]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:44.643 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:44 smithi111 ceph-mon[40056]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:44.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:44 smithi114 ceph-mon[33315]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:46.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:46 smithi114 ceph-mon[33315]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:46 smithi111 ceph-mon[29803]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:46.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:46 smithi111 ceph-mon[40056]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:48.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:48 smithi114 ceph-mon[33315]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:48.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:48 smithi111 ceph-mon[29803]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:48.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:48 smithi111 ceph-mon[40056]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:49.630 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:49.630 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:49.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:49.631 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:49.631 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:49.631 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:49.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:49.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:49.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:50.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:50 smithi111 ceph-mon[29803]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:50.837 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:50 smithi111 ceph-mon[40056]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:50.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:50 smithi114 ceph-mon[33315]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:52 smithi114 ceph-mon[33315]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:52 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[29803]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[40056]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:52 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.637+0000 7f52e77fe640 1 -- 172.21.15.111:0/2231133796 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f52f8063590 con 0x7f52e007ac90 2024-09-16T08:55:53.641 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 1 on host 'smithi111' 2024-09-16T08:55:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f52e007ac90 msgr2=0x7f52e007d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f52e007ac90 0x7f52e007d150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f52ec005fd0 tx=0x7f52ec00b390 comp rx=0 tx=0).stop 2024-09-16T08:55:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a00 msgr2=0x7f52f8198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:55:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a00 0x7f52f8198300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f52e800ea60 tx=0x7f52e800ef20 comp rx=0 tx=0).stop 2024-09-16T08:55:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 shutdown_connections 2024-09-16T08:55:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f52e007ac90 0x7f52e007d150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52f8104520 0x7f52f8199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8103b60 0x7f52f8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 --2- 172.21.15.111:0/2231133796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f8102a00 0x7f52f8198300 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:55:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.639+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 >> 172.21.15.111:0/2231133796 conn(0x7f52f80fe2f0 msgr2=0x7f52f80fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:55:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.640+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 shutdown_connections 2024-09-16T08:55:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:55:53.640+0000 7f52fcdad640 1 -- 172.21.15.111:0/2231133796 wait complete. 2024-09-16T08:55:54.255 DEBUG:teuthology.orchestra.run.smithi111:osd.1> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.1.service 2024-09-16T08:55:54.257 INFO:tasks.cephadm:Deploying osd.2 on smithi111 with /dev/vg_nvme/lv_2... 2024-09-16T08:55:54.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T08:55:54.516 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[29803]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:54.516 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[29803]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-16T08:55:54.516 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:54.516 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:54.516 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:54.517 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[40056]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:54.517 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[40056]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-16T08:55:54.517 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:54.517 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:54.517 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:54 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:54 smithi114 ceph-mon[33315]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:54 smithi114 ceph-mon[33315]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-16T08:55:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:54 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:55:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:54 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:54 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:54.947 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:55:55.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-mon[29803]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T08:55:55.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-mon[29803]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T08:55:55.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-mon[29803]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:55:55.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:55.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-mon[40056]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T08:55:55.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-mon[40056]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T08:55:55.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-mon[40056]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:55:55.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:55.427 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1[49769]: 2024-09-16T08:55:55.256+0000 7fb574517640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T08:55:55.427 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:55:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1[49769]: 2024-09-16T08:55:55.258+0000 7fb57033e640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T08:55:55.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:55 smithi114 ceph-mon[33315]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T08:55:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:55 smithi114 ceph-mon[33315]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T08:55:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:55 smithi114 ceph-mon[33315]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:55:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[29803]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[29803]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:55:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[29803]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T08:55:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[40056]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[40056]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:55:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[40056]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T08:55:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:56.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:56 smithi114 ceph-mon[33315]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:56.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:56 smithi114 ceph-mon[33315]: from='osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:55:56.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:56 smithi114 ceph-mon[33315]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T08:55:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:57.555 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[29803]: purged_snaps scrub starts 2024-09-16T08:55:57.555 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[29803]: purged_snaps scrub ok 2024-09-16T08:55:57.555 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[29803]: osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] boot 2024-09-16T08:55:57.556 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[29803]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T08:55:57.556 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:57.556 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[40056]: purged_snaps scrub starts 2024-09-16T08:55:57.556 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[40056]: purged_snaps scrub ok 2024-09-16T08:55:57.556 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[40056]: osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] boot 2024-09-16T08:55:57.556 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[40056]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T08:55:57.556 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:57 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:57.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:57 smithi114 ceph-mon[33315]: purged_snaps scrub starts 2024-09-16T08:55:57.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:57 smithi114 ceph-mon[33315]: purged_snaps scrub ok 2024-09-16T08:55:57.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:57 smithi114 ceph-mon[33315]: osd.1 [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] boot 2024-09-16T08:55:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:57 smithi114 ceph-mon[33315]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T08:55:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:57 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:55:58.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:58.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T08:55:58.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:55:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:58.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:58.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-16T08:55:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T08:55:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:55:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:55:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:55:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:55:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:55:59.504 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:59 smithi111 ceph-mon[29803]: Detected new or changed devices on smithi111 2024-09-16T08:55:59.504 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:55:59 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi111 to 8032M 2024-09-16T08:55:59.504 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:59 smithi111 ceph-mon[40056]: Detected new or changed devices on smithi111 2024-09-16T08:55:59.504 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:55:59 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi111 to 8032M 2024-09-16T08:55:59.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:59 smithi114 ceph-mon[33315]: Detected new or changed devices on smithi111 2024-09-16T08:55:59.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:55:59 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi111 to 8032M 2024-09-16T08:55:59.891 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:55:59.918 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2024-09-16T08:56:00.166 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:56:00.569 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:00 smithi111 ceph-mon[29803]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:00.569 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:00 smithi111 ceph-mon[40056]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:00.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2854477972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01018e0 msgr2=0x7fdcc0101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:00.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2854477972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01018e0 0x7fdcc0101ce0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009890 tx=0x7fdcb802a380 comp rx=0 tx=0).stop 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2854477972 shutdown_connections 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2854477972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc0103400 0x7fdcc0109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2854477972 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc0102a40 0x7fdcc0102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2854477972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01018e0 0x7fdcc0101ce0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2854477972 >> 172.21.15.111:0/2854477972 conn(0x7fdcc00fd1b0 msgr2=0x7fdcc00ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2854477972 shutdown_connections 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2854477972 wait complete. 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 Processor -- start 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.568+0000 7fdcc83d2640 1 -- start start 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc83d2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01018e0 0x7fdcc019a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc83d2640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc0102a40 0x7fdcc019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:00.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc83d2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc0103400 0x7fdcc019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc83d2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc010db50 con 0x7fdcc01018e0 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc83d2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc010d9d0 con 0x7fdcc0103400 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc83d2640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fdcc010dcd0 con 0x7fdcc0102a40 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc596d640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc0102a40 0x7fdcc019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc616e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01018e0 0x7fdcc019a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc596d640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc0102a40 0x7fdcc019ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:38602/0 (socket says 172.21.15.111:38602) 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc596d640 1 -- 172.21.15.111:0/2524650096 learned_addr learned my addr 172.21.15.111:0/2524650096 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc696f640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc0103400 0x7fdcc019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc616e640 1 -- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc0102a40 msgr2=0x7fdcc019ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc616e640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc0102a40 0x7fdcc019ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc616e640 1 -- 172.21.15.111:0/2524650096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc0103400 msgr2=0x7fdcc019b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc616e640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc0103400 0x7fdcc019b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:00.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc616e640 1 -- 172.21.15.111:0/2524650096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc019c060 con 0x7fdcc01018e0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc616e640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01018e0 0x7fdcc019a530 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdcb802a8a0 tx=0x7fdcb8033d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcbf7fe640 1 -- 172.21.15.111:0/2524650096 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8037040 con 0x7fdcc01018e0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcbf7fe640 1 -- 172.21.15.111:0/2524650096 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fdcb80044a0 con 0x7fdcc01018e0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcbf7fe640 1 -- 172.21.15.111:0/2524650096 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb80035f0 con 0x7fdcc01018e0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.569+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc019c2f0 con 0x7fdcc01018e0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.570+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc01a2860 con 0x7fdcc01018e0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.570+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc0101ce0 con 0x7fdcc01018e0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.571+0000 7fdcbf7fe640 1 -- 172.21.15.111:0/2524650096 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 14) ==== 103024+0+0 (secure 0 0 0) 0x7fdcb8042020 con 0x7fdcc01018e0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.572+0000 7fdcbf7fe640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fdca807ac90 0x7fdca807d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.572+0000 7fdcc596d640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fdca807ac90 0x7fdca807d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.572+0000 7fdcbf7fe640 1 -- 172.21.15.111:0/2524650096 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2199+0+0 (secure 0 0 0) 0x7fdcb80bc150 con 0x7fdcc01018e0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.572+0000 7fdcc596d640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fdca807ac90 0x7fdca807d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4005c80 tx=0x7fdcb4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:56:00.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.576+0000 7fdcbf7fe640 1 -- 172.21.15.111:0/2524650096 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fdcb8049020 con 0x7fdcc01018e0 2024-09-16T08:56:00.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:00 smithi114 ceph-mon[33315]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:00.761+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fdcc010b820 con 0x7fdca807ac90 2024-09-16T08:56:01.419 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:01 smithi111 ceph-mon[40056]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:56:01.420 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:56:01.420 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:56:01.420 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:01 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:01.420 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:01 smithi111 ceph-mon[29803]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:56:01.420 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:56:01.420 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:56:01.420 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:01 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:01.649 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:01 smithi114 ceph-mon[33315]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:56:01.649 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:56:01.649 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:56:01.649 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:01 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:02.411 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:02 smithi111 ceph-mon[29803]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:02.411 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:02 smithi111 ceph-mon[40056]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:02 smithi114 ceph-mon[33315]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1078997443' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7650515a-da82-4064-83a2-d4cb3665f1ba"} : dispatch 2024-09-16T08:56:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1078997443' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7650515a-da82-4064-83a2-d4cb3665f1ba"}]': finished 2024-09-16T08:56:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:03 smithi114 ceph-mon[33315]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T08:56:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:03 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:03.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1078997443' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7650515a-da82-4064-83a2-d4cb3665f1ba"} : dispatch 2024-09-16T08:56:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1078997443' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7650515a-da82-4064-83a2-d4cb3665f1ba"}]': finished 2024-09-16T08:56:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:03 smithi111 ceph-mon[29803]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T08:56:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:03 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1078997443' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7650515a-da82-4064-83a2-d4cb3665f1ba"} : dispatch 2024-09-16T08:56:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1078997443' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7650515a-da82-4064-83a2-d4cb3665f1ba"}]': finished 2024-09-16T08:56:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:03 smithi111 ceph-mon[40056]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T08:56:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:03 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:04 smithi114 ceph-mon[33315]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3523227840' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:56:04.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:04 smithi111 ceph-mon[29803]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:04.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3523227840' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:56:04.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:04 smithi111 ceph-mon[40056]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:04.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3523227840' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:56:06.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:06 smithi114 ceph-mon[33315]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:06.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:06 smithi111 ceph-mon[29803]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:06.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:06 smithi111 ceph-mon[40056]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:08.561 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:08 smithi111 ceph-mon[29803]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:08.562 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:08 smithi111 ceph-mon[40056]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:08.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:08 smithi114 ceph-mon[33315]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:10.541 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:10 smithi111 ceph-mon[29803]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:10.541 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:10 smithi111 ceph-mon[40056]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:10.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:10 smithi114 ceph-mon[33315]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:11.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:11 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-16T08:56:11.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:11 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:11.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:11 smithi114 ceph-mon[33315]: Deploying daemon osd.2 on smithi111 2024-09-16T08:56:11.670 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:11 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-16T08:56:11.670 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:11 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:11.670 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:11 smithi111 ceph-mon[29803]: Deploying daemon osd.2 on smithi111 2024-09-16T08:56:11.670 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:11 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-16T08:56:11.670 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:11 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:11.670 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:11 smithi111 ceph-mon[40056]: Deploying daemon osd.2 on smithi111 2024-09-16T08:56:12.549 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:12 smithi111 ceph-mon[29803]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:12.549 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:12 smithi111 ceph-mon[40056]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:12 smithi114 ceph-mon[33315]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:14.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:14 smithi114 ceph-mon[33315]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:14 smithi111 ceph-mon[40056]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:14 smithi111 ceph-mon[29803]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:16.656 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:16 smithi111 ceph-mon[29803]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:16.657 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:16 smithi111 ceph-mon[40056]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:16.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:16 smithi114 ceph-mon[33315]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:18.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:18 smithi111 ceph-mon[29803]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:18.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:18 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:18.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:18 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:18.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:18 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:18.637 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:18 smithi111 ceph-mon[40056]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:18.637 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:18 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:18.637 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:18 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:18.637 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:18 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:18.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:18 smithi114 ceph-mon[33315]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:18 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:18 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:18 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:20.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:20 smithi111 ceph-mon[29803]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:20.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:20 smithi111 ceph-mon[40056]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:20.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:20 smithi114 ceph-mon[33315]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:22.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:22.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:22.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:22.925 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:22 smithi111 ceph-mon[29803]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:22 smithi111 ceph-mon[40056]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:23.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:22 smithi114 ceph-mon[33315]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:23.486 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 2 on host 'smithi111' 2024-09-16T08:56:23.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.485+0000 7fdcbf7fe640 1 -- 172.21.15.111:0/2524650096 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdcc010b820 con 0x7fdca807ac90 2024-09-16T08:56:23.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fdca807ac90 msgr2=0x7fdca807d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fdca807ac90 0x7fdca807d150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4005c80 tx=0x7fdcb4009260 comp rx=0 tx=0).stop 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01018e0 msgr2=0x7fdcc019a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01018e0 0x7fdcc019a530 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdcb802a8a0 tx=0x7fdcb8033d00 comp rx=0 tx=0).stop 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 shutdown_connections 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fdca807ac90 0x7fdca807d150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc0103400 0x7fdcc019b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc0102a40 0x7fdcc019ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 --2- 172.21.15.111:0/2524650096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01018e0 0x7fdcc019a530 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 >> 172.21.15.111:0/2524650096 conn(0x7fdcc00fd1b0 msgr2=0x7fdcc00fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 shutdown_connections 2024-09-16T08:56:23.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:23.488+0000 7fdcc83d2640 1 -- 172.21.15.111:0/2524650096 wait complete. 2024-09-16T08:56:24.110 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:23 smithi111 ceph-mon[29803]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-16T08:56:24.110 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:23 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:24.110 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:23 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:24.110 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:23 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:24.110 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:23 smithi111 ceph-mon[40056]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-16T08:56:24.110 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:23 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:24.110 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:23 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:24.111 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:23 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:24.141 DEBUG:teuthology.orchestra.run.smithi111:osd.2> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.2.service 2024-09-16T08:56:24.144 INFO:tasks.cephadm:Deploying osd.3 on smithi111 with /dev/vg_nvme/lv_1... 2024-09-16T08:56:24.144 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T08:56:24.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:23 smithi114 ceph-mon[33315]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-16T08:56:24.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:23 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:24.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:23 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:24.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:23 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:25.027 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:56:25.148 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[40056]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:25.149 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[40056]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T08:56:25.149 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[40056]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T08:56:25.149 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[40056]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:56:25.149 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:25.149 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[29803]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:25.149 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[29803]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T08:56:25.149 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[29803]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T08:56:25.150 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[29803]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:56:25.150 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:25.150 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2[55932]: 2024-09-16T08:56:24.867+0000 7f0b78d65640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T08:56:25.150 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2[55932]: 2024-09-16T08:56:24.872+0000 7f0b74b8c640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T08:56:25.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:24 smithi114 ceph-mon[33315]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:25.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:24 smithi114 ceph-mon[33315]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T08:56:25.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:24 smithi114 ceph-mon[33315]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T08:56:25.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:24 smithi114 ceph-mon[33315]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:56:25.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:24 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:26.088 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[29803]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:56:26.088 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[29803]: osdmap e17: 3 total, 2 up, 3 in 2024-09-16T08:56:26.088 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:26.089 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:26.089 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[29803]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:26.089 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[40056]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:56:26.089 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[40056]: osdmap e17: 3 total, 2 up, 3 in 2024-09-16T08:56:26.089 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:26.089 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:26.089 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:25 smithi111 ceph-mon[40056]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:25 smithi114 ceph-mon[33315]: from='osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:56:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:25 smithi114 ceph-mon[33315]: osdmap e17: 3 total, 2 up, 3 in 2024-09-16T08:56:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:25 smithi114 ceph-mon[33315]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-16T08:56:27.011 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[29803]: purged_snaps scrub starts 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[29803]: purged_snaps scrub ok 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[29803]: osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] boot 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[29803]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[40056]: purged_snaps scrub starts 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[40056]: purged_snaps scrub ok 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[40056]: osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] boot 2024-09-16T08:56:27.012 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[40056]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T08:56:27.013 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:26 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:27.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:26 smithi114 ceph-mon[33315]: purged_snaps scrub starts 2024-09-16T08:56:27.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:26 smithi114 ceph-mon[33315]: purged_snaps scrub ok 2024-09-16T08:56:27.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:26 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:27.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:26 smithi114 ceph-mon[33315]: osd.2 [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] boot 2024-09-16T08:56:27.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:26 smithi114 ceph-mon[33315]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T08:56:27.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:26 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:56:28.008 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T08:56:28.008 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: pgmap v79: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:28.008 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-16T08:56:28.008 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: Detected new or changed devices on smithi111 2024-09-16T08:56:28.008 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.008 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi111 to 5354M 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: pgmap v79: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-16T08:56:28.009 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: Detected new or changed devices on smithi111 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi111 to 5354M 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:28.010 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:27 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T08:56:28.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: pgmap v79: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: Detected new or changed devices on smithi111 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi111 to 5354M 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:27 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:29.026 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:28 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T08:56:29.026 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:28 smithi111 ceph-mon[40056]: osdmap e20: 3 total, 3 up, 3 in 2024-09-16T08:56:29.026 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:28 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-16T08:56:29.027 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:28 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T08:56:29.027 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:28 smithi111 ceph-mon[29803]: osdmap e20: 3 total, 3 up, 3 in 2024-09-16T08:56:29.027 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:28 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-16T08:56:29.027 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59858]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T08:56:29.027 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.027 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59858]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.027 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59858]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.027 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:56:28 smithi111 sudo[59852]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T08:56:29.028 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:56:28 smithi111 sudo[59852]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.028 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:56:28 smithi111 sudo[59852]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.028 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:56:28 smithi111 sudo[59852]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.028 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:56:28 smithi111 sudo[59855]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T08:56:29.028 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:56:28 smithi111 sudo[59855]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.028 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:56:28 smithi111 sudo[59855]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.028 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 08:56:28 smithi111 sudo[59855]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:28 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T08:56:29.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:28 smithi114 ceph-mon[33315]: osdmap e20: 3 total, 3 up, 3 in 2024-09-16T08:56:29.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:28 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-16T08:56:29.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 sudo[35275]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T08:56:29.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 sudo[35275]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 sudo[35275]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.151 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 sudo[35275]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 sudo[35278]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T08:56:29.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 sudo[35278]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 sudo[35278]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 sudo[35278]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59876]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T08:56:29.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59876]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59876]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59876]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59879]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T08:56:29.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59879]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59879]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59879]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.392 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59867]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T08:56:29.392 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59867]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.392 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59867]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.392 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59867]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.392 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59882]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T08:56:29.392 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59882]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.392 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59882]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.392 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59882]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.392 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59885]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T08:56:29.393 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59885]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.393 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59885]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.393 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59885]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.393 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59870]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T08:56:29.393 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59870]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.393 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59870]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.393 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59870]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.393 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59873]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T08:56:29.393 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59873]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T08:56:29.393 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59873]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T08:56:29.393 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 08:56:29 smithi111 sudo[59873]: pam_unix(sudo:session): session closed for user root 2024-09-16T08:56:29.394 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:56:29.420 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_1 2024-09-16T08:56:29.663 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:56:30.036 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T08:56:30.036 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:30.036 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: osdmap e21: 3 total, 3 up, 3 in 2024-09-16T08:56:30.036 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:30.036 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T08:56:30.036 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T08:56:30.036 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:56:30.036 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T08:56:30.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[40056]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: osdmap e21: 3 total, 3 up, 3 in 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:56:30.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:56:30.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T08:56:30.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:30.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T08:56:30.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:56:30.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:56:30.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:56:30.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T08:56:30.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:29 smithi111 ceph-mon[29803]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T08:56:30.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 -- 172.21.15.111:0/4090582406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e00103440 msgr2=0x7f9e00109dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 --2- 172.21.15.111:0/4090582406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e00103440 0x7f9e00109dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9df8009930 tx=0x7f9df802a420 comp rx=0 tx=0).stop 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 -- 172.21.15.111:0/4090582406 shutdown_connections 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 --2- 172.21.15.111:0/4090582406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e00103440 0x7f9e00109dc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 --2- 172.21.15.111:0/4090582406 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e00102a80 0x7f9e00102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 --2- 172.21.15.111:0/4090582406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e00101920 0x7f9e00101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 -- 172.21.15.111:0/4090582406 >> 172.21.15.111:0/4090582406 conn(0x7f9e000fd1d0 msgr2=0x7f9e000ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 -- 172.21.15.111:0/4090582406 shutdown_connections 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 -- 172.21.15.111:0/4090582406 wait complete. 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 Processor -- start 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.120+0000 7f9e05a92640 1 -- start start 2024-09-16T08:56:30.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9e05a92640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e00101920 0x7f9e00197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9e05a92640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e00102a80 0x7f9e00197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9e05a92640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e00103440 0x7f9e001984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9e05a92640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0010ca40 con 0x7f9e00103440 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9e05a92640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0010c8c0 con 0x7f9e00102a80 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9e05a92640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0010cbc0 con 0x7f9e00101920 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dfffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e00103440 0x7f9e001984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dff7fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e00101920 0x7f9e00197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dfffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e00103440 0x7f9e001984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57668/0 (socket says 172.21.15.111:57668) 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dfffff640 1 -- 172.21.15.111:0/1634834345 learned_addr learned my addr 172.21.15.111:0/1634834345 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dfeffd640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e00102a80 0x7f9e00197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dff7fe640 1 -- 172.21.15.111:0/1634834345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e00102a80 msgr2=0x7f9e00197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:30.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dff7fe640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e00102a80 0x7f9e00197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dff7fe640 1 -- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e00103440 msgr2=0x7f9e001984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dff7fe640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e00103440 0x7f9e001984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dff7fe640 1 -- 172.21.15.111:0/1634834345 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e00198d70 con 0x7f9e00101920 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dfeffd640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e00102a80 0x7f9e00197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dff7fe640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e00101920 0x7f9e00197240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e9f0 tx=0x7f9df000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dfcff9640 1 -- 172.21.15.111:0/1634834345 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df000cd80 con 0x7f9e00101920 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e00199060 con 0x7f9e00101920 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dfcff9640 1 -- 172.21.15.111:0/1634834345 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f9df00118c0 con 0x7f9e00101920 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9dfcff9640 1 -- 172.21.15.111:0/1634834345 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df00107b0 con 0x7f9e00101920 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.121+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e0006b630 con 0x7f9e00101920 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.123+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e0010d200 con 0x7f9e00101920 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.123+0000 7f9dfcff9640 1 -- 172.21.15.111:0/1634834345 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 14) ==== 103024+0+0 (secure 0 0 0) 0x7f9df0011470 con 0x7f9e00101920 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.124+0000 7f9dfcff9640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f9de807ac90 0x7f9de807d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:30.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.124+0000 7f9dfeffd640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f9de807ac90 0x7f9de807d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:30.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.124+0000 7f9dfcff9640 1 -- 172.21.15.111:0/1634834345 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 3017+0+0 (secure 0 0 0) 0x7f9df009d540 con 0x7f9e00101920 2024-09-16T08:56:30.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.124+0000 7f9dfeffd640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f9de807ac90 0x7f9de807d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009c20 tx=0x7f9df4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:56:30.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.128+0000 7f9dfcff9640 1 -- 172.21.15.111:0/1634834345 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f9df0016030 con 0x7f9e00101920 2024-09-16T08:56:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T08:56:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: osdmap e21: 3 total, 3 up, 3 in 2024-09-16T08:56:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T08:56:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:56:30.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:56:30.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T08:56:30.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:29 smithi114 ceph-mon[33315]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T08:56:30.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:30.315+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9e0006b8f0 con 0x7f9de807ac90 2024-09-16T08:56:31.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[29803]: osdmap e22: 3 total, 3 up, 3 in 2024-09-16T08:56:31.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[29803]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:56:31.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:56:31.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:56:31.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:31.115 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[40056]: osdmap e22: 3 total, 3 up, 3 in 2024-09-16T08:56:31.115 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[40056]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:56:31.115 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:56:31.115 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:56:31.116 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:30 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:31.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:30 smithi114 ceph-mon[33315]: osdmap e22: 3 total, 3 up, 3 in 2024-09-16T08:56:31.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:30 smithi114 ceph-mon[33315]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:56:31.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:56:31.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:56:31.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:30 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:31.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:31.916+0000 7f9dfcff9640 1 -- 172.21.15.111:0/1634834345 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f9df0061190 con 0x7f9e00101920 2024-09-16T08:56:32.028 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:31 smithi111 ceph-mon[29803]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:32.028 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:31 smithi111 ceph-mon[29803]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T08:56:32.028 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:31 smithi111 ceph-mon[29803]: Cluster is now healthy 2024-09-16T08:56:32.028 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:31 smithi111 ceph-mon[40056]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:32.028 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:31 smithi111 ceph-mon[40056]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T08:56:32.028 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:31 smithi111 ceph-mon[40056]: Cluster is now healthy 2024-09-16T08:56:32.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:31 smithi114 ceph-mon[33315]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:32.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:31 smithi114 ceph-mon[33315]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T08:56:32.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:31 smithi114 ceph-mon[33315]: Cluster is now healthy 2024-09-16T08:56:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[29803]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-16T08:56:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3491891287' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4e778b16-cf6a-405a-8ae3-899c7b7988b5"} : dispatch 2024-09-16T08:56:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3491891287' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e778b16-cf6a-405a-8ae3-899c7b7988b5"}]': finished 2024-09-16T08:56:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[29803]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T08:56:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:33.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[40056]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-16T08:56:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3491891287' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4e778b16-cf6a-405a-8ae3-899c7b7988b5"} : dispatch 2024-09-16T08:56:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3491891287' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e778b16-cf6a-405a-8ae3-899c7b7988b5"}]': finished 2024-09-16T08:56:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[40056]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T08:56:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:32 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:33.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:32 smithi114 ceph-mon[33315]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-16T08:56:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3491891287' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "4e778b16-cf6a-405a-8ae3-899c7b7988b5"} : dispatch 2024-09-16T08:56:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3491891287' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e778b16-cf6a-405a-8ae3-899c7b7988b5"}]': finished 2024-09-16T08:56:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:32 smithi114 ceph-mon[33315]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T08:56:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:32 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:33 smithi111 ceph-mon[29803]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1060872839' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:56:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:33 smithi111 ceph-mon[40056]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1060872839' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:56:34.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:33 smithi114 ceph-mon[33315]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:34.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1060872839' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:56:36.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:35 smithi114 ceph-mon[33315]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:36.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:35 smithi111 ceph-mon[40056]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:36.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:35 smithi111 ceph-mon[29803]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:37 smithi111 ceph-mon[29803]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:38.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:37 smithi111 ceph-mon[40056]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:38.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:37 smithi114 ceph-mon[33315]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:39 smithi111 ceph-mon[29803]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:40.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:39 smithi111 ceph-mon[40056]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:40.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:39 smithi114 ceph-mon[33315]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:41.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:40 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-16T08:56:41.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:40 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:41.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:40 smithi111 ceph-mon[29803]: Deploying daemon osd.3 on smithi111 2024-09-16T08:56:41.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:40 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-16T08:56:41.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:40 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:41.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:40 smithi111 ceph-mon[40056]: Deploying daemon osd.3 on smithi111 2024-09-16T08:56:41.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:40 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-16T08:56:41.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:40 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:41.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:40 smithi114 ceph-mon[33315]: Deploying daemon osd.3 on smithi111 2024-09-16T08:56:42.036 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:41 smithi111 ceph-mon[29803]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:42.037 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:41 smithi111 ceph-mon[40056]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:42.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:41 smithi114 ceph-mon[33315]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:43 smithi111 ceph-mon[29803]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:43 smithi111 ceph-mon[40056]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:44.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:43 smithi114 ceph-mon[33315]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:46.381 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:45 smithi111 ceph-mon[29803]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:46.381 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:45 smithi111 ceph-mon[40056]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:46.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:45 smithi114 ceph-mon[33315]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:48.212 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:47 smithi111 ceph-mon[29803]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:48.212 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:47 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:48.212 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:47 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:48.212 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:47 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:48.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:47 smithi111 ceph-mon[40056]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:48.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:47 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:48.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:47 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:48.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:47 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:48.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:47 smithi114 ceph-mon[33315]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:48.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:47 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:48.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:47 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:48.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:47 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:49 smithi111 ceph-mon[29803]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:50.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:49 smithi111 ceph-mon[40056]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:50.215 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:49 smithi114 ceph-mon[33315]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:51.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:51.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:51.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:51.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:51.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:52.786 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:52 smithi111 ceph-mon[29803]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:52.786 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:52 smithi111 ceph-mon[40056]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:52.825 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 3 on host 'smithi111' 2024-09-16T08:56:52.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.824+0000 7f9dfcff9640 1 -- 172.21.15.111:0/1634834345 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9e0006b8f0 con 0x7f9de807ac90 2024-09-16T08:56:52.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f9de807ac90 msgr2=0x7f9de807d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:52.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f9de807ac90 0x7f9de807d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009c20 tx=0x7f9df4005f90 comp rx=0 tx=0).stop 2024-09-16T08:56:52.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e00101920 msgr2=0x7f9e00197240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:52.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e00101920 0x7f9e00197240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e9f0 tx=0x7f9df000eeb0 comp rx=0 tx=0).stop 2024-09-16T08:56:52.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 shutdown_connections 2024-09-16T08:56:52.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f9de807ac90 0x7f9de807d150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:52.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e00103440 0x7f9e001984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:52.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e00102a80 0x7f9e00197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:52.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 --2- 172.21.15.111:0/1634834345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e00101920 0x7f9e00197240 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:52.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.828+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 >> 172.21.15.111:0/1634834345 conn(0x7f9e000fd1d0 msgr2=0x7f9e000feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:56:52.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.829+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 shutdown_connections 2024-09-16T08:56:52.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:56:52.829+0000 7f9e05a92640 1 -- 172.21.15.111:0/1634834345 wait complete. 2024-09-16T08:56:52.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:52 smithi114 ceph-mon[33315]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:53.500 DEBUG:teuthology.orchestra.run.smithi111:osd.3> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.3.service 2024-09-16T08:56:53.502 INFO:tasks.cephadm:Deploying osd.4 on smithi114 with /dev/vg_nvme/lv_4... 2024-09-16T08:56:53.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T08:56:53.740 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:56:53.791 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[29803]: from='osd.3 [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T08:56:53.791 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[29803]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T08:56:53.791 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:53.791 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:53.791 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:53.791 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[40056]: from='osd.3 [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T08:56:53.791 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[40056]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T08:56:53.792 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:53.792 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:53.792 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:53.860 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:53 smithi114 ceph-mon[33315]: from='osd.3 [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T08:56:53.860 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:53 smithi114 ceph-mon[33315]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-16T08:56:53.861 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:56:53.861 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:53.861 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[29803]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[29803]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T08:56:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[29803]: from='osd.3 [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:56:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[29803]: osdmap e24: 4 total, 3 up, 4 in 2024-09-16T08:56:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[29803]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:56:54.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[40056]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[40056]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T08:56:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[40056]: from='osd.3 [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:56:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[40056]: osdmap e24: 4 total, 3 up, 4 in 2024-09-16T08:56:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-mon[40056]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:56:54.807 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:54 smithi114 ceph-mon[33315]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:54.807 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:54 smithi114 ceph-mon[33315]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T08:56:54.807 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:54 smithi114 ceph-mon[33315]: from='osd.3 [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:56:54.807 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:54 smithi114 ceph-mon[33315]: osdmap e24: 4 total, 3 up, 4 in 2024-09-16T08:56:54.807 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:54 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:54.807 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:54 smithi114 ceph-mon[33315]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]} : dispatch 2024-09-16T08:56:55.117 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3[62019]: 2024-09-16T08:56:54.895+0000 7f0986f11640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T08:56:55.117 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 08:56:54 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3[62019]: 2024-09-16T08:56:54.901+0000 7f097ed30640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:55 smithi114 ceph-mon[33315]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:55 smithi114 ceph-mon[33315]: osdmap e25: 4 total, 3 up, 4 in 2024-09-16T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:55 smithi114 ceph-mon[33315]: from='osd.3 ' entity='osd.3' 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[40056]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[40056]: osdmap e25: 4 total, 3 up, 4 in 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[40056]: from='osd.3 ' entity='osd.3' 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[29803]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[29803]: osdmap e25: 4 total, 3 up, 4 in 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:55 smithi111 ceph-mon[29803]: from='osd.3 ' entity='osd.3' 2024-09-16T08:56:56.484 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:56:56.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-09-16T08:56:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: purged_snaps scrub starts 2024-09-16T08:56:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: purged_snaps scrub ok 2024-09-16T08:56:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: osd.3 [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] boot 2024-09-16T08:56:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: osdmap e26: 4 total, 4 up, 4 in 2024-09-16T08:56:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: purged_snaps scrub starts 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: purged_snaps scrub ok 2024-09-16T08:56:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: osd.3 [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] boot 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: osdmap e26: 4 total, 4 up, 4 in 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.749 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:56:56.869 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: purged_snaps scrub starts 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: purged_snaps scrub ok 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: osd.3 [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] boot 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: osdmap e26: 4 total, 4 up, 4 in 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T08:56:56.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:56.871 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:56.871 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:56:56.871 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:56:57.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.175+0000 7f8bda5c6640 1 -- 172.21.15.114:0/735919772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4102a00 msgr2=0x7f8bd4102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:57.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.175+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/735919772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4102a00 0x7f8bd4102e00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009930 tx=0x7f8bc802a420 comp rx=0 tx=0).stop 2024-09-16T08:56:57.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.175+0000 7f8bda5c6640 1 -- 172.21.15.114:0/735919772 shutdown_connections 2024-09-16T08:56:57.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.175+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/735919772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bd4104520 0x7f8bd410aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:57.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.175+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/735919772 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8bd4103b60 0x7f8bd4103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.175+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/735919772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4102a00 0x7f8bd4102e00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.175+0000 7f8bda5c6640 1 -- 172.21.15.114:0/735919772 >> 172.21.15.114:0/735919772 conn(0x7f8bd40fe2f0 msgr2=0x7f8bd4100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.175+0000 7f8bda5c6640 1 -- 172.21.15.114:0/735919772 shutdown_connections 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.175+0000 7f8bda5c6640 1 -- 172.21.15.114:0/735919772 wait complete. 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bda5c6640 1 Processor -- start 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bda5c6640 1 -- start start 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bda5c6640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8bd4102a00 0x7f8bd41982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bda5c6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4103b60 0x7f8bd4198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bda5c6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bd4104520 0x7f8bd4199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bda5c6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd410db20 con 0x7f8bd4104520 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bda5c6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd410d9a0 con 0x7f8bd4103b60 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bda5c6640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd410dca0 con 0x7f8bd4102a00 2024-09-16T08:56:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4103b60 0x7f8bd4198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4103b60 0x7f8bd4198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40372/0 (socket says 172.21.15.114:40372) 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd37fe640 1 -- 172.21.15.114:0/4076189096 learned_addr learned my addr 172.21.15.114:0/4076189096 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd3fff640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8bd4102a00 0x7f8bd41982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd8b63640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bd4104520 0x7f8bd4199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd37fe640 1 -- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8bd4102a00 msgr2=0x7f8bd41982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd37fe640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8bd4102a00 0x7f8bd41982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd37fe640 1 -- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bd4104520 msgr2=0x7f8bd4199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd37fe640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bd4104520 0x7f8bd4199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd37fe640 1 -- 172.21.15.114:0/4076189096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd4199e10 con 0x7f8bd4103b60 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.176+0000 7f8bd37fe640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4103b60 0x7f8bd4198930 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400c920 tx=0x7f8bc400cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.177+0000 7f8bd17fa640 1 -- 172.21.15.114:0/4076189096 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4007c70 con 0x7f8bd4103b60 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.177+0000 7f8bd17fa640 1 -- 172.21.15.114:0/4076189096 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f8bc4007e10 con 0x7f8bd4103b60 2024-09-16T08:56:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.177+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd419a100 con 0x7f8bd4103b60 2024-09-16T08:56:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.177+0000 7f8bd17fa640 1 -- 172.21.15.114:0/4076189096 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4018660 con 0x7f8bd4103b60 2024-09-16T08:56:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.177+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd41a07e0 con 0x7f8bd4103b60 2024-09-16T08:56:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.178+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd4102e00 con 0x7f8bd4103b60 2024-09-16T08:56:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.178+0000 7f8bd17fa640 1 -- 172.21.15.114:0/4076189096 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f8bc400f450 con 0x7f8bd4103b60 2024-09-16T08:56:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.179+0000 7f8bd17fa640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f8bbc07ad20 0x7f8bbc07d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:56:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.179+0000 7f8bd17fa640 1 -- 172.21.15.114:0/4076189096 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3449+0+0 (secure 0 0 0) 0x7f8bc409d4b0 con 0x7f8bd4103b60 2024-09-16T08:56:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.179+0000 7f8bd3fff640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f8bbc07ad20 0x7f8bbc07d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:56:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.179+0000 7f8bd3fff640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f8bbc07ad20 0x7f8bbc07d1e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8006010 tx=0x7f8bc8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:56:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.183+0000 7f8bd17fa640 1 -- 172.21.15.114:0/4076189096 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f8bc4014030 con 0x7f8bd4103b60 2024-09-16T08:56:57.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:56:57.366+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8bd4063590 con 0x7f8bbc07ad20 2024-09-16T08:56:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:57 smithi114 ceph-mon[33315]: Detected new or changed devices on smithi111 2024-09-16T08:56:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:57 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi111 to 4016M 2024-09-16T08:56:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:57 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:56:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:57 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:56:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:57 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:57.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[29803]: Detected new or changed devices on smithi111 2024-09-16T08:56:57.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi111 to 4016M 2024-09-16T08:56:57.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:56:57.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:56:57.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:57.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[40056]: Detected new or changed devices on smithi111 2024-09-16T08:56:57.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi111 to 4016M 2024-09-16T08:56:57.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:56:57.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:56:57.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:57 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:56:58.773 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:58 smithi114 ceph-mon[33315]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:56:58.773 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:58 smithi114 ceph-mon[33315]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:56:58.773 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:56:58 smithi114 ceph-mon[33315]: osdmap e27: 4 total, 4 up, 4 in 2024-09-16T08:56:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:58 smithi111 ceph-mon[29803]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:56:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:58 smithi111 ceph-mon[29803]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:56:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:56:58 smithi111 ceph-mon[29803]: osdmap e27: 4 total, 4 up, 4 in 2024-09-16T08:56:58.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:58 smithi111 ceph-mon[40056]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:56:58.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:58 smithi111 ceph-mon[40056]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:56:58.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:56:58 smithi111 ceph-mon[40056]: osdmap e27: 4 total, 4 up, 4 in 2024-09-16T08:57:00.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:00 smithi114 ceph-mon[33315]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:00.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/1531763964' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a15db99-3151-4039-81a5-d651e5b5b756"} : dispatch 2024-09-16T08:57:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a15db99-3151-4039-81a5-d651e5b5b756"} : dispatch 2024-09-16T08:57:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a15db99-3151-4039-81a5-d651e5b5b756"}]': finished 2024-09-16T08:57:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:00 smithi114 ceph-mon[33315]: osdmap e28: 5 total, 4 up, 5 in 2024-09-16T08:57:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:00 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/3563695075' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:57:00.728 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[29803]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:00.728 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/1531763964' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a15db99-3151-4039-81a5-d651e5b5b756"} : dispatch 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a15db99-3151-4039-81a5-d651e5b5b756"} : dispatch 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a15db99-3151-4039-81a5-d651e5b5b756"}]': finished 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[29803]: osdmap e28: 5 total, 4 up, 5 in 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/3563695075' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[40056]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/1531763964' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a15db99-3151-4039-81a5-d651e5b5b756"} : dispatch 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2a15db99-3151-4039-81a5-d651e5b5b756"} : dispatch 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a15db99-3151-4039-81a5-d651e5b5b756"}]': finished 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[40056]: osdmap e28: 5 total, 4 up, 5 in 2024-09-16T08:57:00.729 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:00.730 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/3563695075' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:57:02.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:02 smithi114 ceph-mon[33315]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:02 smithi111 ceph-mon[29803]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:02.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:02 smithi111 ceph-mon[40056]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:04.759 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:04 smithi114 ceph-mon[33315]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:04 smithi111 ceph-mon[29803]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:04.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:04 smithi111 ceph-mon[40056]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:06.561 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:06 smithi114 ceph-mon[33315]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:06 smithi111 ceph-mon[29803]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:06.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:06 smithi111 ceph-mon[40056]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:07.909 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:07 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-16T08:57:07.909 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:07 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:07.909 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:07 smithi114 ceph-mon[33315]: Deploying daemon osd.4 on smithi114 2024-09-16T08:57:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:07 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-16T08:57:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:07 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:07 smithi111 ceph-mon[29803]: Deploying daemon osd.4 on smithi114 2024-09-16T08:57:07.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:07 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-16T08:57:07.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:07 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:07.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:07 smithi111 ceph-mon[40056]: Deploying daemon osd.4 on smithi114 2024-09-16T08:57:08.595 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:08 smithi114 ceph-mon[33315]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:08 smithi111 ceph-mon[29803]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:08.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:08 smithi111 ceph-mon[40056]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:10.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:10 smithi114 ceph-mon[33315]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:10.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:10 smithi111 ceph-mon[40056]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:10 smithi111 ceph-mon[29803]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:12.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:12 smithi114 ceph-mon[33315]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:12 smithi111 ceph-mon[29803]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:12.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:12 smithi111 ceph-mon[40056]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:14.779 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:14 smithi114 ceph-mon[33315]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:14.779 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:14 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:14.779 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:14 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:14.779 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:14 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:14 smithi111 ceph-mon[29803]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:14 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:14 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:14 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:14.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:14 smithi111 ceph-mon[40056]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:14.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:14 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:14.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:14 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:14.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:14 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:16 smithi114 ceph-mon[33315]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:16.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:16 smithi111 ceph-mon[29803]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:16.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:16 smithi111 ceph-mon[40056]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:17.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:17 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:17 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:17 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:17 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:17 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:17.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:17 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:18 smithi114 ceph-mon[33315]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:18 smithi111 ceph-mon[29803]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:18.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:18 smithi111 ceph-mon[40056]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:19.235 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 4 on host 'smithi114' 2024-09-16T08:57:19.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.234+0000 7f8bd17fa640 1 -- 172.21.15.114:0/4076189096 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8bd4063590 con 0x7f8bbc07ad20 2024-09-16T08:57:19.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f8bbc07ad20 msgr2=0x7f8bbc07d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:19.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f8bbc07ad20 0x7f8bbc07d1e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8006010 tx=0x7f8bc8000bc0 comp rx=0 tx=0).stop 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4103b60 msgr2=0x7f8bd4198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4103b60 0x7f8bd4198930 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400c920 tx=0x7f8bc400cde0 comp rx=0 tx=0).stop 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 shutdown_connections 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f8bbc07ad20 0x7f8bbc07d1e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bd4104520 0x7f8bd4199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bd4103b60 0x7f8bd4198930 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 --2- 172.21.15.114:0/4076189096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8bd4102a00 0x7f8bd41982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.236+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 >> 172.21.15.114:0/4076189096 conn(0x7f8bd40fe2f0 msgr2=0x7f8bd40fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.237+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 shutdown_connections 2024-09-16T08:57:19.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:19.237+0000 7f8bda5c6640 1 -- 172.21.15.114:0/4076189096 wait complete. 2024-09-16T08:57:19.812 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:19 smithi114 ceph-mon[33315]: from='osd.4 [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T08:57:19.812 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:19 smithi114 ceph-mon[33315]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T08:57:19.812 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:19 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:19.812 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:19 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:19.812 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:19 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:19.839 DEBUG:teuthology.orchestra.run.smithi114:osd.4> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.4.service 2024-09-16T08:57:19.843 INFO:tasks.cephadm:Deploying osd.5 on smithi114 with /dev/vg_nvme/lv_3... 2024-09-16T08:57:19.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T08:57:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[29803]: from='osd.4 [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T08:57:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[29803]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T08:57:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[40056]: from='osd.4 [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T08:57:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[40056]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-16T08:57:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:19 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:20.570 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:57:21.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:20 smithi114 ceph-mon[33315]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:21.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:20 smithi114 ceph-mon[33315]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T08:57:21.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:20 smithi114 ceph-mon[33315]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T08:57:21.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:20 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:21.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:20 smithi114 ceph-mon[33315]: from='osd.4 [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:21.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:20 smithi114 ceph-mon[33315]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:21.011 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 08:57:20 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T08:57:20.686+0000 7f8ef4115640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T08:57:21.011 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 08:57:20 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T08:57:20.689+0000 7f8eeff3c640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T08:57:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[29803]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[29803]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T08:57:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[29803]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T08:57:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[29803]: from='osd.4 [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[29803]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:21.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[40056]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:21.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[40056]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T08:57:21.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[40056]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T08:57:21.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:21.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[40056]: from='osd.4 [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:21.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:20 smithi111 ceph-mon[40056]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:21.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:21 smithi114 ceph-mon[33315]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:57:21.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:21 smithi114 ceph-mon[33315]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T08:57:21.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:21.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:21 smithi111 ceph-mon[29803]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:57:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:21 smithi111 ceph-mon[29803]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T08:57:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:22.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:21 smithi111 ceph-mon[40056]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:57:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:21 smithi111 ceph-mon[40056]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T08:57:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:22.982 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:22 smithi114 ceph-mon[33315]: purged_snaps scrub starts 2024-09-16T08:57:22.982 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:22 smithi114 ceph-mon[33315]: purged_snaps scrub ok 2024-09-16T08:57:22.982 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:22 smithi114 ceph-mon[33315]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:22.982 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:22 smithi114 ceph-mon[33315]: osd.4 [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] boot 2024-09-16T08:57:22.983 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:22 smithi114 ceph-mon[33315]: osdmap e31: 5 total, 5 up, 5 in 2024-09-16T08:57:22.983 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:22 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:22.983 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:22 smithi114 ceph-mon[33315]: osdmap e32: 5 total, 5 up, 5 in 2024-09-16T08:57:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[29803]: purged_snaps scrub starts 2024-09-16T08:57:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[29803]: purged_snaps scrub ok 2024-09-16T08:57:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[29803]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[29803]: osd.4 [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] boot 2024-09-16T08:57:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[29803]: osdmap e31: 5 total, 5 up, 5 in 2024-09-16T08:57:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:23.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[29803]: osdmap e32: 5 total, 5 up, 5 in 2024-09-16T08:57:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[40056]: purged_snaps scrub starts 2024-09-16T08:57:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[40056]: purged_snaps scrub ok 2024-09-16T08:57:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[40056]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-16T08:57:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[40056]: osd.4 [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] boot 2024-09-16T08:57:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[40056]: osdmap e31: 5 total, 5 up, 5 in 2024-09-16T08:57:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:57:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:22 smithi111 ceph-mon[40056]: osdmap e32: 5 total, 5 up, 5 in 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: Detected new or changed devices on smithi114 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi114 to 17088M 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:24 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:25.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:25.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T08:57:25.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: Detected new or changed devices on smithi114 2024-09-16T08:57:25.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:25.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:25.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:57:25.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi114 to 17088M 2024-09-16T08:57:25.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: Detected new or changed devices on smithi114 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi114 to 17088M 2024-09-16T08:57:25.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:25.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:25.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:25.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:24 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:25.233 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:57:25.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-09-16T08:57:25.518 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:57:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.966+0000 7fe579362640 1 -- 172.21.15.114:0/3023979177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574102a00 msgr2=0x7fe574102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.966+0000 7fe579362640 1 --2- 172.21.15.114:0/3023979177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574102a00 0x7fe574102e00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe564009930 tx=0x7fe56402a420 comp rx=0 tx=0).stop 2024-09-16T08:57:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.966+0000 7fe579362640 1 -- 172.21.15.114:0/3023979177 shutdown_connections 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.966+0000 7fe579362640 1 --2- 172.21.15.114:0/3023979177 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe574104520 0x7fe57410aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.966+0000 7fe579362640 1 --2- 172.21.15.114:0/3023979177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe574103b60 0x7fe574103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.966+0000 7fe579362640 1 --2- 172.21.15.114:0/3023979177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574102a00 0x7fe574102e00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.966+0000 7fe579362640 1 -- 172.21.15.114:0/3023979177 >> 172.21.15.114:0/3023979177 conn(0x7fe5740fe2f0 msgr2=0x7fe574100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.966+0000 7fe579362640 1 -- 172.21.15.114:0/3023979177 shutdown_connections 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.966+0000 7fe579362640 1 -- 172.21.15.114:0/3023979177 wait complete. 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe579362640 1 Processor -- start 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe579362640 1 -- start start 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe579362640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe574102a00 0x7fe574198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe579362640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574103b60 0x7fe5741989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe579362640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe574104520 0x7fe5741995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:57:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe579362640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe57410dad0 con 0x7fe574102a00 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe579362640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe57410d950 con 0x7fe574103b60 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe579362640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fe57410dc50 con 0x7fe574104520 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe56bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574103b60 0x7fe5741989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe56bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574103b60 0x7fe5741989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52086/0 (socket says 172.21.15.114:52086) 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe56bfff640 1 -- 172.21.15.114:0/174677289 learned_addr learned my addr 172.21.15.114:0/174677289 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe572ffd640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe574102a00 0x7fe574198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.967+0000 7fe5737fe640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe574104520 0x7fe5741995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe56bfff640 1 -- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe574104520 msgr2=0x7fe5741995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe56bfff640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe574104520 0x7fe5741995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe56bfff640 1 -- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe574102a00 msgr2=0x7fe574198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe56bfff640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe574102a00 0x7fe574198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe56bfff640 1 -- 172.21.15.114:0/174677289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe574199e80 con 0x7fe574103b60 2024-09-16T08:57:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe56bfff640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574103b60 0x7fe5741989a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00d950 tx=0x7fe55c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe570ff9640 1 -- 172.21.15.114:0/174677289 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c017070 con 0x7fe574103b60 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe570ff9640 1 -- 172.21.15.114:0/174677289 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fe55c009c30 con 0x7fe574103b60 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57419a170 con 0x7fe574103b60 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe570ff9640 1 -- 172.21.15.114:0/174677289 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c00f3c0 con 0x7fe574103b60 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.968+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5741a0850 con 0x7fe574103b60 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.969+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe574102e00 con 0x7fe574103b60 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.970+0000 7fe570ff9640 1 -- 172.21.15.114:0/174677289 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7fe55c010460 con 0x7fe574103b60 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.970+0000 7fe570ff9640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe56007ad20 0x7fe56007d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.971+0000 7fe570ff9640 1 -- 172.21.15.114:0/174677289 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 3991+0+0 (secure 0 0 0) 0x7fe55c09d190 con 0x7fe574103b60 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.971+0000 7fe572ffd640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe56007ad20 0x7fe56007d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.971+0000 7fe572ffd640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe56007ad20 0x7fe56007d1e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe5640057c0 tx=0x7fe5640058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:57:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:25.975+0000 7fe570ff9640 1 -- 172.21.15.114:0/174677289 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fe55c0691a0 con 0x7fe574103b60 2024-09-16T08:57:26.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:26.163+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe57402e8f0 con 0x7fe56007ad20 2024-09-16T08:57:26.878 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:26 smithi114 ceph-mon[33315]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:26.878 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:26 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:57:26.878 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:26 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:57:26.878 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:26 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:27.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:26 smithi111 ceph-mon[29803]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:27.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:26 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:57:27.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:26 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:57:27.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:26 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:27.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:26 smithi111 ceph-mon[40056]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:27.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:26 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:57:27.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:26 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:57:27.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:26 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:27.761 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:27 smithi114 ceph-mon[33315]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:57:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:27 smithi111 ceph-mon[29803]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:57:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:27 smithi111 ceph-mon[40056]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:57:28.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:28 smithi114 ceph-mon[33315]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/877562395' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f16b15-1806-477e-96d5-2f84ab4cc399"} : dispatch 2024-09-16T08:57:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f16b15-1806-477e-96d5-2f84ab4cc399"} : dispatch 2024-09-16T08:57:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45f16b15-1806-477e-96d5-2f84ab4cc399"}]': finished 2024-09-16T08:57:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:28 smithi114 ceph-mon[33315]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T08:57:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:28 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[29803]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/877562395' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f16b15-1806-477e-96d5-2f84ab4cc399"} : dispatch 2024-09-16T08:57:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f16b15-1806-477e-96d5-2f84ab4cc399"} : dispatch 2024-09-16T08:57:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45f16b15-1806-477e-96d5-2f84ab4cc399"}]': finished 2024-09-16T08:57:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[29803]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T08:57:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[40056]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/877562395' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f16b15-1806-477e-96d5-2f84ab4cc399"} : dispatch 2024-09-16T08:57:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f16b15-1806-477e-96d5-2f84ab4cc399"} : dispatch 2024-09-16T08:57:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45f16b15-1806-477e-96d5-2f84ab4cc399"}]': finished 2024-09-16T08:57:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[40056]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T08:57:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:28 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:30.077 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/2235037129' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:57:30.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/2235037129' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:57:30.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/2235037129' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:57:31.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:30 smithi114 ceph-mon[33315]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-16T08:57:31.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:30 smithi111 ceph-mon[29803]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-16T08:57:31.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:30 smithi111 ceph-mon[40056]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-16T08:57:32.854 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:32 smithi114 ceph-mon[33315]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T08:57:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:32 smithi111 ceph-mon[29803]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T08:57:33.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:32 smithi111 ceph-mon[40056]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T08:57:35.088 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:34 smithi114 ceph-mon[33315]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-16T08:57:35.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:34 smithi111 ceph-mon[29803]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-16T08:57:35.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:34 smithi111 ceph-mon[40056]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-16T08:57:36.036 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:35 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-16T08:57:36.036 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:35 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:35 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-16T08:57:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:35 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:36.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:35 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-16T08:57:36.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:35 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:36.931 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:36 smithi114 ceph-mon[33315]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-16T08:57:36.931 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:36 smithi114 ceph-mon[33315]: Deploying daemon osd.5 on smithi114 2024-09-16T08:57:37.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:36 smithi111 ceph-mon[29803]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-16T08:57:37.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:36 smithi111 ceph-mon[29803]: Deploying daemon osd.5 on smithi114 2024-09-16T08:57:37.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:36 smithi111 ceph-mon[40056]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-16T08:57:37.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:36 smithi111 ceph-mon[40056]: Deploying daemon osd.5 on smithi114 2024-09-16T08:57:38.833 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:38 smithi114 ceph-mon[33315]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-16T08:57:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:38 smithi111 ceph-mon[29803]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-16T08:57:39.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:38 smithi111 ceph-mon[40056]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-16T08:57:41.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:40 smithi114 ceph-mon[33315]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:41.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:40 smithi111 ceph-mon[29803]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:41.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:40 smithi111 ceph-mon[40056]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:42.747 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:42 smithi114 ceph-mon[33315]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:42 smithi111 ceph-mon[29803]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:42 smithi111 ceph-mon[40056]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:43.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:43 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:43.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:43 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:43.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:43 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:43 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:43 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:43 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:43 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:43 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:43 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:44.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:44 smithi114 ceph-mon[33315]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:45.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:44 smithi111 ceph-mon[29803]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:45.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:44 smithi111 ceph-mon[40056]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[29803]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[40056]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:46 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:46.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:46 smithi114 ceph-mon[33315]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:46.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:46 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:46.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:46 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:46 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:46 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:46 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:48.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.288+0000 7fe570ff9640 1 -- 172.21.15.114:0/174677289 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe57402e8f0 con 0x7fe56007ad20 2024-09-16T08:57:48.289 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 5 on host 'smithi114' 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe56007ad20 msgr2=0x7fe56007d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe56007ad20 0x7fe56007d1e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe5640057c0 tx=0x7fe5640058d0 comp rx=0 tx=0).stop 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574103b60 msgr2=0x7fe5741989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574103b60 0x7fe5741989a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00d950 tx=0x7fe55c00de10 comp rx=0 tx=0).stop 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 shutdown_connections 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe56007ad20 0x7fe56007d1e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe574104520 0x7fe5741995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe574103b60 0x7fe5741989a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 --2- 172.21.15.114:0/174677289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe574102a00 0x7fe574198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 >> 172.21.15.114:0/174677289 conn(0x7fe5740fe2f0 msgr2=0x7fe5740fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:57:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 shutdown_connections 2024-09-16T08:57:48.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:48.291+0000 7fe579362640 1 -- 172.21.15.114:0/174677289 wait complete. 2024-09-16T08:57:48.600 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:48 smithi114 ceph-mon[33315]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:48.600 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:48 smithi114 ceph-mon[33315]: from='osd.5 [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T08:57:48.600 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:48 smithi114 ceph-mon[33315]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T08:57:48.600 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:48 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:48.600 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:48 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:48.600 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:48 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:48.901 DEBUG:teuthology.orchestra.run.smithi114:osd.5> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.5.service 2024-09-16T08:57:48.904 INFO:tasks.cephadm:Deploying osd.6 on smithi114 with /dev/vg_nvme/lv_2... 2024-09-16T08:57:48.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T08:57:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[29803]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[29803]: from='osd.5 [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T08:57:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[29803]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T08:57:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:48.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[40056]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:48.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[40056]: from='osd.5 [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T08:57:48.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[40056]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-16T08:57:48.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:57:48.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:48.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:48 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:49.643 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:57:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:49 smithi114 ceph-mon[33315]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T08:57:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:49 smithi114 ceph-mon[33315]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T08:57:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:49 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:49 smithi114 ceph-mon[33315]: from='osd.5 [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:49 smithi114 ceph-mon[33315]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:49.665 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 08:57:49 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T08:57:49.445+0000 7f89a78a5640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T08:57:49.665 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 08:57:49 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T08:57:49.449+0000 7f89a36cc640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T08:57:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[29803]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T08:57:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[29803]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T08:57:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[29803]: from='osd.5 [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[29803]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[40056]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T08:57:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[40056]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T08:57:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[40056]: from='osd.5 [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:49 smithi111 ceph-mon[40056]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:57:50.878 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:50 smithi114 ceph-mon[33315]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:50.878 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:50 smithi114 ceph-mon[33315]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:57:50.878 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:50 smithi114 ceph-mon[33315]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T08:57:50.878 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:50 smithi111 ceph-mon[29803]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:50 smithi111 ceph-mon[29803]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:57:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:50 smithi111 ceph-mon[29803]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T08:57:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:50 smithi111 ceph-mon[40056]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-16T08:57:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:50 smithi111 ceph-mon[40056]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:57:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:50 smithi111 ceph-mon[40056]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T08:57:50.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:51.750 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:51 smithi114 ceph-mon[33315]: purged_snaps scrub starts 2024-09-16T08:57:51.750 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:51 smithi114 ceph-mon[33315]: purged_snaps scrub ok 2024-09-16T08:57:51.750 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:51 smithi114 ceph-mon[33315]: osd.5 [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] boot 2024-09-16T08:57:51.750 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:51 smithi114 ceph-mon[33315]: osdmap e37: 6 total, 6 up, 6 in 2024-09-16T08:57:51.750 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:51 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[29803]: purged_snaps scrub starts 2024-09-16T08:57:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[29803]: purged_snaps scrub ok 2024-09-16T08:57:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[29803]: osd.5 [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] boot 2024-09-16T08:57:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[29803]: osdmap e37: 6 total, 6 up, 6 in 2024-09-16T08:57:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[40056]: purged_snaps scrub starts 2024-09-16T08:57:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[40056]: purged_snaps scrub ok 2024-09-16T08:57:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[40056]: osd.5 [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] boot 2024-09-16T08:57:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[40056]: osdmap e37: 6 total, 6 up, 6 in 2024-09-16T08:57:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:51 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:57:52.562 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:52 smithi114 ceph-mon[33315]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:52.562 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:52 smithi114 ceph-mon[33315]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T08:57:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:52 smithi111 ceph-mon[29803]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:52 smithi111 ceph-mon[29803]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T08:57:52.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:52 smithi111 ceph-mon[40056]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:52.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:52 smithi111 ceph-mon[40056]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T08:57:53.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T08:57:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: Detected new or changed devices on smithi114 2024-09-16T08:57:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:57:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:57:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:53 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T08:57:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: Detected new or changed devices on smithi114 2024-09-16T08:57:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:57:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:57:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: Detected new or changed devices on smithi114 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:57:53.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:53 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:57:54.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:54 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi114 to 8544M 2024-09-16T08:57:54.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:54 smithi114 ceph-mon[33315]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:54.507 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:57:54.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-09-16T08:57:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:57:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:54 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi114 to 8544M 2024-09-16T08:57:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:54 smithi111 ceph-mon[29803]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:54.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:54 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi114 to 8544M 2024-09-16T08:57:54.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:54 smithi111 ceph-mon[40056]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.221+0000 7f58fac0a640 1 -- 172.21.15.114:0/3712827632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 msgr2=0x7f58f4103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.221+0000 7f58fac0a640 1 --2- 172.21.15.114:0/3712827632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 0x7f58f4103fe0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009930 tx=0x7f58e402a420 comp rx=0 tx=0).stop 2024-09-16T08:57:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.221+0000 7f58fac0a640 1 -- 172.21.15.114:0/3712827632 shutdown_connections 2024-09-16T08:57:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.221+0000 7f58fac0a640 1 --2- 172.21.15.114:0/3712827632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f4104520 0x7f58f410aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.221+0000 7f58fac0a640 1 --2- 172.21.15.114:0/3712827632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 0x7f58f4103fe0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.221+0000 7f58fac0a640 1 --2- 172.21.15.114:0/3712827632 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58f4102a00 0x7f58f4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.221+0000 7f58fac0a640 1 -- 172.21.15.114:0/3712827632 >> 172.21.15.114:0/3712827632 conn(0x7f58f40fe2f0 msgr2=0x7f58f4100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:57:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 -- 172.21.15.114:0/3712827632 shutdown_connections 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 -- 172.21.15.114:0/3712827632 wait complete. 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 Processor -- start 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 -- start start 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58f4102a00 0x7f58f41983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 0x7f58f4198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f4104520 0x7f58f4199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f58f410dba0 con 0x7f58f4104520 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f58f410da20 con 0x7f58f4103b60 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58fac0a640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f58f410dd20 con 0x7f58f4102a00 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58ebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 0x7f58f4198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58ebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 0x7f58f4198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51454/0 (socket says 172.21.15.114:51454) 2024-09-16T08:57:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.222+0000 7f58ebfff640 1 -- 172.21.15.114:0/1761631765 learned_addr learned my addr 172.21.15.114:0/1761631765 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58ebfff640 1 -- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58f4102a00 msgr2=0x7f58f41983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58f91a7640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f4104520 0x7f58f4199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58f89a6640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58f4102a00 0x7f58f41983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58ebfff640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58f4102a00 0x7f58f41983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58ebfff640 1 -- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f4104520 msgr2=0x7f58f4199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58ebfff640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f4104520 0x7f58f4199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58ebfff640 1 -- 172.21.15.114:0/1761631765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f4199f00 con 0x7f58f4103b60 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58ebfff640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 0x7f58f4198a20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f58e402a940 tx=0x7f58e40052b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58e9ffb640 1 -- 172.21.15.114:0/1761631765 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e403f070 con 0x7f58f4103b60 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58e9ffb640 1 -- 172.21.15.114:0/1761631765 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f58e4003e50 con 0x7f58f4103b60 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f419a190 con 0x7f58f4103b60 2024-09-16T08:57:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58e9ffb640 1 -- 172.21.15.114:0/1761631765 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4035030 con 0x7f58f4103b60 2024-09-16T08:57:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.223+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f41a08d0 con 0x7f58f4103b60 2024-09-16T08:57:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.224+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f4102e00 con 0x7f58f4103b60 2024-09-16T08:57:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.225+0000 7f58e9ffb640 1 -- 172.21.15.114:0/1761631765 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f58e4042070 con 0x7f58f4103b60 2024-09-16T08:57:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.225+0000 7f58e9ffb640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f58dc07ad20 0x7f58dc07d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:57:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.225+0000 7f58e9ffb640 1 -- 172.21.15.114:0/1761631765 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4423+0+0 (secure 0 0 0) 0x7f58e40be300 con 0x7f58f4103b60 2024-09-16T08:57:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.226+0000 7f58f89a6640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f58dc07ad20 0x7f58dc07d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:57:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.226+0000 7f58f89a6640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f58dc07ad20 0x7f58dc07d1e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f58f000a810 tx=0x7f58f0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:57:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.230+0000 7f58e9ffb640 1 -- 172.21.15.114:0/1761631765 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f58e4085e00 con 0x7f58f4103b60 2024-09-16T08:57:55.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:57:55.414+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f58f4063590 con 0x7f58dc07ad20 2024-09-16T08:57:55.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:57:55.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:57:55.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:55.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:57:55.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:57:55.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:57:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:57:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:57:56.728 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:56 smithi114 ceph-mon[33315]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:56.728 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:56 smithi114 ceph-mon[33315]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:57:56.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:56 smithi111 ceph-mon[29803]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:56.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:56 smithi111 ceph-mon[29803]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:57:56.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:56 smithi111 ceph-mon[40056]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:56.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:56 smithi111 ceph-mon[40056]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:57:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:58 smithi114 ceph-mon[33315]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/806959750' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "321fe560-27fd-4b30-a298-ae609af8eec0"} : dispatch 2024-09-16T08:57:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "321fe560-27fd-4b30-a298-ae609af8eec0"} : dispatch 2024-09-16T08:57:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "321fe560-27fd-4b30-a298-ae609af8eec0"}]': finished 2024-09-16T08:57:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:58 smithi114 ceph-mon[33315]: osdmap e40: 7 total, 6 up, 7 in 2024-09-16T08:57:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:57:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:57:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/2679480236' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:57:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[29803]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/806959750' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "321fe560-27fd-4b30-a298-ae609af8eec0"} : dispatch 2024-09-16T08:57:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "321fe560-27fd-4b30-a298-ae609af8eec0"} : dispatch 2024-09-16T08:57:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "321fe560-27fd-4b30-a298-ae609af8eec0"}]': finished 2024-09-16T08:57:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[29803]: osdmap e40: 7 total, 6 up, 7 in 2024-09-16T08:57:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:57:58.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/2679480236' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:57:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[40056]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:57:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/806959750' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "321fe560-27fd-4b30-a298-ae609af8eec0"} : dispatch 2024-09-16T08:57:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "321fe560-27fd-4b30-a298-ae609af8eec0"} : dispatch 2024-09-16T08:57:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "321fe560-27fd-4b30-a298-ae609af8eec0"}]': finished 2024-09-16T08:57:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[40056]: osdmap e40: 7 total, 6 up, 7 in 2024-09-16T08:57:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:57:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:57:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/2679480236' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:58:00.728 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:00 smithi111 ceph-mon[29803]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:00.728 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:00 smithi111 ceph-mon[40056]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:00.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:00 smithi114 ceph-mon[33315]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:02.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:02 smithi114 ceph-mon[33315]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:02 smithi111 ceph-mon[29803]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:02.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:02 smithi111 ceph-mon[40056]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:04 smithi114 ceph-mon[33315]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:04 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-16T08:58:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:04 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:04 smithi111 ceph-mon[29803]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:04 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-16T08:58:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:04 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:04.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:04 smithi111 ceph-mon[40056]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:04.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:04 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-16T08:58:04.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:04 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:05.888 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:05 smithi114 ceph-mon[33315]: Deploying daemon osd.6 on smithi114 2024-09-16T08:58:05.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:05 smithi111 ceph-mon[29803]: Deploying daemon osd.6 on smithi114 2024-09-16T08:58:05.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:05 smithi111 ceph-mon[40056]: Deploying daemon osd.6 on smithi114 2024-09-16T08:58:06.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:06 smithi114 ceph-mon[33315]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:06.925 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:06 smithi111 ceph-mon[29803]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:06.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:06 smithi111 ceph-mon[40056]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:08.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:08 smithi114 ceph-mon[33315]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:08.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:08 smithi111 ceph-mon[40056]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:08 smithi111 ceph-mon[29803]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:10.727 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:10 smithi114 ceph-mon[33315]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:10.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:10 smithi111 ceph-mon[40056]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:10 smithi111 ceph-mon[29803]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:12.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:12 smithi114 ceph-mon[33315]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:12.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:12 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:12.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:12 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:12.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:12 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:12 smithi111 ceph-mon[29803]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:12 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:12 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:12 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:12.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:12 smithi111 ceph-mon[40056]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:12.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:12 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:12.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:12 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:12.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:12 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:14.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:14 smithi114 ceph-mon[33315]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:14.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:14 smithi111 ceph-mon[40056]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:14 smithi111 ceph-mon[29803]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:16.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:15 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:16.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:15 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:16.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:15 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:16.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:15 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:16.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:15 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:16.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:16.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:16.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:16.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:16.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:15 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:16.981 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:16 smithi114 ceph-mon[33315]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:17.021 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 6 on host 'smithi114' 2024-09-16T08:58:17.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.020+0000 7f58e9ffb640 1 -- 172.21.15.114:0/1761631765 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f58f4063590 con 0x7f58dc07ad20 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f58dc07ad20 msgr2=0x7f58dc07d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f58dc07ad20 0x7f58dc07d1e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f58f000a810 tx=0x7f58f0008040 comp rx=0 tx=0).stop 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 msgr2=0x7f58f4198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 0x7f58f4198a20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f58e402a940 tx=0x7f58e40052b0 comp rx=0 tx=0).stop 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 shutdown_connections 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f58dc07ad20 0x7f58dc07d1e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f4104520 0x7f58f4199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58f4103b60 0x7f58f4198a20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 --2- 172.21.15.114:0/1761631765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58f4102a00 0x7f58f41983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 >> 172.21.15.114:0/1761631765 conn(0x7f58f40fe2f0 msgr2=0x7f58f40fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 shutdown_connections 2024-09-16T08:58:17.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:17.022+0000 7f58fac0a640 1 -- 172.21.15.114:0/1761631765 wait complete. 2024-09-16T08:58:17.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:16 smithi111 ceph-mon[29803]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:17.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:16 smithi111 ceph-mon[40056]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:17.587 DEBUG:teuthology.orchestra.run.smithi114:osd.6> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.6.service 2024-09-16T08:58:17.589 INFO:tasks.cephadm:Deploying osd.7 on smithi114 with /dev/vg_nvme/lv_1... 2024-09-16T08:58:17.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T08:58:17.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:17 smithi114 ceph-mon[33315]: from='osd.6 [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T08:58:17.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:17 smithi114 ceph-mon[33315]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T08:58:17.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:17 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:17.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:17 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:17.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:17 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:18.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[29803]: from='osd.6 [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T08:58:18.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[29803]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T08:58:18.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:18.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:18.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:18.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[40056]: from='osd.6 [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T08:58:18.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[40056]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-16T08:58:18.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:18.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:18.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:17 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:18.645 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:58:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:18 smithi114 ceph-mon[33315]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:18 smithi114 ceph-mon[33315]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T08:58:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:18 smithi114 ceph-mon[33315]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T08:58:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:18 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:58:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:18 smithi114 ceph-mon[33315]: from='osd.6 [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:18 smithi114 ceph-mon[33315]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:19.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[29803]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:19.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[29803]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T08:58:19.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[29803]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T08:58:19.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:58:19.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[29803]: from='osd.6 [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:19.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[29803]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:19.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[40056]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:19.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[40056]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T08:58:19.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[40056]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T08:58:19.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:58:19.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[40056]: from='osd.6 [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:19.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:18 smithi111 ceph-mon[40056]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:19.663 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T08:58:19.174+0000 7fbe90187640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T08:58:19.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T08:58:19.179+0000 7fbe87fa6640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T08:58:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-mon[33315]: purged_snaps scrub starts 2024-09-16T08:58:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-mon[33315]: purged_snaps scrub ok 2024-09-16T08:58:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-mon[33315]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:58:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-mon[33315]: osdmap e42: 7 total, 6 up, 7 in 2024-09-16T08:58:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:58:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-mon[33315]: from='osd.6 ' entity='osd.6' 2024-09-16T08:58:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-mon[33315]: osd.6 [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] boot 2024-09-16T08:58:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-mon[33315]: osdmap e43: 7 total, 7 up, 7 in 2024-09-16T08:58:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:19 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:58:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[29803]: purged_snaps scrub starts 2024-09-16T08:58:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[29803]: purged_snaps scrub ok 2024-09-16T08:58:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[29803]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:58:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[29803]: osdmap e42: 7 total, 6 up, 7 in 2024-09-16T08:58:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:58:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[29803]: from='osd.6 ' entity='osd.6' 2024-09-16T08:58:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[29803]: osd.6 [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] boot 2024-09-16T08:58:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[29803]: osdmap e43: 7 total, 7 up, 7 in 2024-09-16T08:58:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:58:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[40056]: purged_snaps scrub starts 2024-09-16T08:58:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[40056]: purged_snaps scrub ok 2024-09-16T08:58:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[40056]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:58:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[40056]: osdmap e42: 7 total, 6 up, 7 in 2024-09-16T08:58:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:58:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[40056]: from='osd.6 ' entity='osd.6' 2024-09-16T08:58:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[40056]: osd.6 [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] boot 2024-09-16T08:58:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[40056]: osdmap e43: 7 total, 7 up, 7 in 2024-09-16T08:58:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:19 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:58:20.880 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:20 smithi114 ceph-mon[33315]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:20.881 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:20 smithi114 ceph-mon[33315]: osdmap e44: 7 total, 7 up, 7 in 2024-09-16T08:58:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:20 smithi111 ceph-mon[29803]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:20 smithi111 ceph-mon[29803]: osdmap e44: 7 total, 7 up, 7 in 2024-09-16T08:58:21.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:20 smithi111 ceph-mon[40056]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-16T08:58:21.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:20 smithi111 ceph-mon[40056]: osdmap e44: 7 total, 7 up, 7 in 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: Detected new or changed devices on smithi114 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi114 to 5696M 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:22.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:22.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:21 smithi114 ceph-mon[33315]: osdmap e45: 7 total, 7 up, 7 in 2024-09-16T08:58:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: Detected new or changed devices on smithi114 2024-09-16T08:58:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi114 to 5696M 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[29803]: osdmap e45: 7 total, 7 up, 7 in 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: Detected new or changed devices on smithi114 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:22.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:22.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi114 to 5696M 2024-09-16T08:58:22.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:22.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:22.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:22.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:21 smithi111 ceph-mon[40056]: osdmap e45: 7 total, 7 up, 7 in 2024-09-16T08:58:22.541 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-16T08:58:22.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-09-16T08:58:22.810 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:58:23.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 -- 172.21.15.114:0/2348637685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c101920 msgr2=0x7f4b8c101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/2348637685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c101920 0x7f4b8c101d20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009930 tx=0x7f4b7c02a420 comp rx=0 tx=0).stop 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 -- 172.21.15.114:0/2348637685 shutdown_connections 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/2348637685 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b8c103440 0x7f4b8c109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/2348637685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c102a80 0x7f4b8c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/2348637685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c101920 0x7f4b8c101d20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 -- 172.21.15.114:0/2348637685 >> 172.21.15.114:0/2348637685 conn(0x7f4b8c0fd1d0 msgr2=0x7f4b8c0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 -- 172.21.15.114:0/2348637685 shutdown_connections 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 -- 172.21.15.114:0/2348637685 wait complete. 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.232+0000 7f4b91aa2640 1 Processor -- start 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b91aa2640 1 -- start start 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b91aa2640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b8c101920 0x7f4b8c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b91aa2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c102a80 0x7f4b8c1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b91aa2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c103440 0x7f4b8c199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b91aa2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c10db50 con 0x7f4b8c103440 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b91aa2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c10d9d0 con 0x7f4b8c102a80 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b91aa2640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f4b8c10dcd0 con 0x7f4b8c101920 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c102a80 0x7f4b8c1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c102a80 0x7f4b8c1989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41822/0 (socket says 172.21.15.114:41822) 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8affd640 1 -- 172.21.15.114:0/344005459 learned_addr learned my addr 172.21.15.114:0/344005459 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8b7fe640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b8c101920 0x7f4b8c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8bfff640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c103440 0x7f4b8c199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8affd640 1 -- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b8c101920 msgr2=0x7f4b8c198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8affd640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b8c101920 0x7f4b8c198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8affd640 1 -- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c103440 msgr2=0x7f4b8c199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8affd640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c103440 0x7f4b8c199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8affd640 1 -- 172.21.15.114:0/344005459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8c199ea0 con 0x7f4b8c102a80 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8bfff640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c103440 0x7f4b8c199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b8affd640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c102a80 0x7f4b8c1989c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000c920 tx=0x7f4b8000cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.233+0000 7f4b88ff9640 1 -- 172.21.15.114:0/344005459 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80007c70 con 0x7f4b8c102a80 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.234+0000 7f4b88ff9640 1 -- 172.21.15.114:0/344005459 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f4b80007e10 con 0x7f4b8c102a80 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.234+0000 7f4b88ff9640 1 -- 172.21.15.114:0/344005459 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80018660 con 0x7f4b8c102a80 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.234+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8c19a190 con 0x7f4b8c102a80 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.234+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b8c1a0870 con 0x7f4b8c102a80 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.234+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b8c101d20 con 0x7f4b8c102a80 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.235+0000 7f4b88ff9640 1 -- 172.21.15.114:0/344005459 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f4b8000f450 con 0x7f4b8c102a80 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.235+0000 7f4b88ff9640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7407ad20 0x7f4b7407d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.236+0000 7f4b88ff9640 1 -- 172.21.15.114:0/344005459 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 4855+0+0 (secure 0 0 0) 0x7f4b8009d9c0 con 0x7f4b8c102a80 2024-09-16T08:58:23.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.236+0000 7f4b8b7fe640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7407ad20 0x7f4b7407d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:23.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.236+0000 7f4b8b7fe640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7407ad20 0x7f4b7407d1e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009900 tx=0x7f4b7c0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:23.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.240+0000 7f4b88ff9640 1 -- 172.21.15.114:0/344005459 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f4b8009fe20 con 0x7f4b8c102a80 2024-09-16T08:58:23.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:23.425+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f4b8c02e880 con 0x7f4b7407ad20 2024-09-16T08:58:24.057 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:23 smithi114 ceph-mon[33315]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:24.057 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:23 smithi114 ceph-mon[33315]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:58:24.057 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:23 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:58:24.057 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:23 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:58:24.057 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:23 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:24.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[29803]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:24.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[29803]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:58:24.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:58:24.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:58:24.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:24.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[40056]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:24.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[40056]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:58:24.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-16T08:58:24.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-16T08:58:24.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:23 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:26.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:25 smithi114 ceph-mon[33315]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:26.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/4204570283' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "16273109-6547-46b3-b670-6173327b7677"} : dispatch 2024-09-16T08:58:26.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "16273109-6547-46b3-b670-6173327b7677"} : dispatch 2024-09-16T08:58:26.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16273109-6547-46b3-b670-6173327b7677"}]': finished 2024-09-16T08:58:26.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:25 smithi114 ceph-mon[33315]: osdmap e46: 8 total, 7 up, 8 in 2024-09-16T08:58:26.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:26.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[29803]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:26.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/4204570283' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "16273109-6547-46b3-b670-6173327b7677"} : dispatch 2024-09-16T08:58:26.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "16273109-6547-46b3-b670-6173327b7677"} : dispatch 2024-09-16T08:58:26.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16273109-6547-46b3-b670-6173327b7677"}]': finished 2024-09-16T08:58:26.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[29803]: osdmap e46: 8 total, 7 up, 8 in 2024-09-16T08:58:26.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:26.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[40056]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:26.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/4204570283' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "16273109-6547-46b3-b670-6173327b7677"} : dispatch 2024-09-16T08:58:26.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "16273109-6547-46b3-b670-6173327b7677"} : dispatch 2024-09-16T08:58:26.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16273109-6547-46b3-b670-6173327b7677"}]': finished 2024-09-16T08:58:26.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[40056]: osdmap e46: 8 total, 7 up, 8 in 2024-09-16T08:58:26.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:27.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/2095948456' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:58:27.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/2095948456' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:58:27.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/2095948456' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-16T08:58:28.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:27 smithi114 ceph-mon[33315]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:28.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:27 smithi111 ceph-mon[29803]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:28.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:27 smithi111 ceph-mon[40056]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:30.228 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:29 smithi111 ceph-mon[29803]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:30.228 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:29 smithi111 ceph-mon[40056]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:30.314 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:29 smithi114 ceph-mon[33315]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:32.149 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:31 smithi114 ceph-mon[33315]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:32.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:31 smithi111 ceph-mon[29803]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:32.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:31 smithi111 ceph-mon[40056]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:34.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:33 smithi114 ceph-mon[33315]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:33 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-16T08:58:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:33 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:33 smithi114 ceph-mon[33315]: Deploying daemon osd.7 on smithi114 2024-09-16T08:58:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:33 smithi111 ceph-mon[29803]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:33 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-16T08:58:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:33 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:33 smithi111 ceph-mon[29803]: Deploying daemon osd.7 on smithi114 2024-09-16T08:58:34.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:33 smithi111 ceph-mon[40056]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:34.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:33 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-16T08:58:34.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:33 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:34.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:33 smithi111 ceph-mon[40056]: Deploying daemon osd.7 on smithi114 2024-09-16T08:58:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:35 smithi114 ceph-mon[33315]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:36.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:35 smithi111 ceph-mon[29803]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:36.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:35 smithi111 ceph-mon[40056]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:38.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:37 smithi114 ceph-mon[33315]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:38.425 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:37 smithi111 ceph-mon[29803]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:38.425 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:37 smithi111 ceph-mon[40056]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:40.241 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:39 smithi114 ceph-mon[33315]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:39 smithi111 ceph-mon[29803]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:40.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:39 smithi111 ceph-mon[40056]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:41.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:40 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:41.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:40 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:41.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:40 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:40 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:40 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:40 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:41.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:40 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:41.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:40 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:41.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:40 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:42.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:42 smithi114 ceph-mon[33315]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:42 smithi111 ceph-mon[29803]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:42.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:42 smithi111 ceph-mon[40056]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:44.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:43 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:44.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:43 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:44.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:43 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:43 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:43 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:43 smithi114 ceph-mon[33315]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[29803]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:43 smithi111 ceph-mon[40056]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:45.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.886+0000 7f4b88ff9640 1 -- 172.21.15.114:0/344005459 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4b8c02e880 con 0x7f4b7407ad20 2024-09-16T08:58:45.888 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 7 on host 'smithi114' 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7407ad20 msgr2=0x7f4b7407d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7407ad20 0x7f4b7407d1e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009900 tx=0x7f4b7c0058d0 comp rx=0 tx=0).stop 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c102a80 msgr2=0x7f4b8c1989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c102a80 0x7f4b8c1989c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000c920 tx=0x7f4b8000cde0 comp rx=0 tx=0).stop 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 shutdown_connections 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7407ad20 0x7f4b7407d1e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c103440 0x7f4b8c199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b8c102a80 0x7f4b8c1989c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 --2- 172.21.15.114:0/344005459 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b8c101920 0x7f4b8c198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:45.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 >> 172.21.15.114:0/344005459 conn(0x7f4b8c0fd1d0 msgr2=0x7f4b8c0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 shutdown_connections 2024-09-16T08:58:45.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:45.889+0000 7f4b91aa2640 1 -- 172.21.15.114:0/344005459 wait complete. 2024-09-16T08:58:46.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:45 smithi114 ceph-mon[33315]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:46.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:45 smithi114 ceph-mon[33315]: from='osd.7 [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T08:58:46.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:45 smithi114 ceph-mon[33315]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T08:58:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:45 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:45 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:45 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[29803]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[29803]: from='osd.7 [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T08:58:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[29803]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T08:58:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[40056]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[40056]: from='osd.7 [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T08:58:46.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[40056]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-16T08:58:46.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:46.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:46.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:45 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:46.538 DEBUG:teuthology.orchestra.run.smithi114:osd.7> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.7.service 2024-09-16T08:58:46.540 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-16T08:58:46.541 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T08:58:46.790 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:58:47.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:46 smithi114 ceph-mon[33315]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T08:58:47.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:46 smithi114 ceph-mon[33315]: from='osd.7 [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:46 smithi114 ceph-mon[33315]: osdmap e47: 8 total, 7 up, 8 in 2024-09-16T08:58:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:46 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:46 smithi114 ceph-mon[33315]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[29803]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T08:58:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[29803]: from='osd.7 [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[29803]: osdmap e47: 8 total, 7 up, 8 in 2024-09-16T08:58:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[29803]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[40056]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T08:58:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[40056]: from='osd.7 [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[40056]: osdmap e47: 8 total, 7 up, 8 in 2024-09-16T08:58:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:46 smithi111 ceph-mon[40056]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-16T08:58:47.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 -- 172.21.15.111:0/1114656765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c104510 msgr2=0x7f5e0c10ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:47.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 --2- 172.21.15.111:0/1114656765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c104510 0x7f5e0c10ae90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009c10 tx=0x7f5e0402a6e0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 -- 172.21.15.111:0/1114656765 shutdown_connections 2024-09-16T08:58:47.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 --2- 172.21.15.111:0/1114656765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c104510 0x7f5e0c10ae90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 --2- 172.21.15.111:0/1114656765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5e0c103b50 0x7f5e0c103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 --2- 172.21.15.111:0/1114656765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e0c1029f0 0x7f5e0c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 -- 172.21.15.111:0/1114656765 >> 172.21.15.111:0/1114656765 conn(0x7f5e0c0fe2c0 msgr2=0x7f5e0c100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 -- 172.21.15.111:0/1114656765 shutdown_connections 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 -- 172.21.15.111:0/1114656765 wait complete. 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.310+0000 7f5e11420640 1 Processor -- start 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e11420640 1 -- start start 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e11420640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c1029f0 0x7f5e0c1982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e11420640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e0c103b50 0x7f5e0c198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e11420640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5e0c104510 0x7f5e0c199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e11420640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c10dae0 con 0x7f5e0c1029f0 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e11420640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c10d960 con 0x7f5e0c103b50 2024-09-16T08:58:47.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e11420640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f5e0c10dc60 con 0x7f5e0c104510 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0affd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c1029f0 0x7f5e0c1982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0affd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c1029f0 0x7f5e0c1982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60560/0 (socket says 172.21.15.111:60560) 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0affd640 1 -- 172.21.15.111:0/3170166743 learned_addr learned my addr 172.21.15.111:0/3170166743 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0a7fc640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e0c103b50 0x7f5e0c198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0b7fe640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5e0c104510 0x7f5e0c199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0affd640 1 -- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5e0c104510 msgr2=0x7f5e0c199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0affd640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5e0c104510 0x7f5e0c199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0affd640 1 -- 172.21.15.111:0/3170166743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e0c103b50 msgr2=0x7f5e0c198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0affd640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e0c103b50 0x7f5e0c198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0affd640 1 -- 172.21.15.111:0/3170166743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0c199e10 con 0x7f5e0c1029f0 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.311+0000 7f5e0affd640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c1029f0 0x7f5e0c1982e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00d950 tx=0x7f5dfc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.312+0000 7f5e03fff640 1 -- 172.21.15.111:0/3170166743 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc017030 con 0x7f5e0c1029f0 2024-09-16T08:58:47.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.312+0000 7f5e03fff640 1 -- 172.21.15.111:0/3170166743 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f5dfc004020 con 0x7f5e0c1029f0 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.312+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0c19a100 con 0x7f5e0c1029f0 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.312+0000 7f5e03fff640 1 -- 172.21.15.111:0/3170166743 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc005430 con 0x7f5e0c1029f0 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.312+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e0c1a07e0 con 0x7f5e0c1029f0 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.313+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e0c102df0 con 0x7f5e0c1029f0 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.313+0000 7f5e03fff640 1 -- 172.21.15.111:0/3170166743 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f5dfc009a30 con 0x7f5e0c1029f0 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.314+0000 7f5e03fff640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5df807ad20 0x7f5df807d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.314+0000 7f5e0a7fc640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5df807ad20 0x7f5df807d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.314+0000 7f5e03fff640 1 -- 172.21.15.111:0/3170166743 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5007+0+0 (secure 0 0 0) 0x7f5dfc09d360 con 0x7f5e0c1029f0 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.314+0000 7f5e0a7fc640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5df807ad20 0x7f5df807d1e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5df40096c0 tx=0x7f5df4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:47.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.318+0000 7f5e03fff640 1 -- 172.21.15.111:0/3170166743 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f5dfc068f80 con 0x7f5e0c1029f0 2024-09-16T08:58:47.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.496+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f5e0c02e5c0 con 0x7f5e0c1029f0 2024-09-16T08:58:47.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.497+0000 7f5e03fff640 1 -- 172.21.15.111:0/3170166743 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v48) ==== 74+0+130 (secure 0 0 0) 0x7f5dfc06de30 con 0x7f5e0c1029f0 2024-09-16T08:58:47.498 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:58:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5df807ad20 msgr2=0x7f5df807d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5df807ad20 0x7f5df807d1e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5df40096c0 tx=0x7f5df4009260 comp rx=0 tx=0).stop 2024-09-16T08:58:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c1029f0 msgr2=0x7f5e0c1982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c1029f0 0x7f5e0c1982e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00d950 tx=0x7f5dfc00de10 comp rx=0 tx=0).stop 2024-09-16T08:58:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 shutdown_connections 2024-09-16T08:58:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5df807ad20 0x7f5df807d1e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5e0c104510 0x7f5e0c199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e0c103b50 0x7f5e0c198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 --2- 172.21.15.111:0/3170166743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e0c1029f0 0x7f5e0c1982e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:47.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 >> 172.21.15.111:0/3170166743 conn(0x7f5e0c0fe2c0 msgr2=0x7f5e0c0fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:47.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 shutdown_connections 2024-09-16T08:58:47.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:47.499+0000 7f5e11420640 1 -- 172.21.15.111:0/3170166743 wait complete. 2024-09-16T08:58:47.533 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 08:58:47 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T08:58:47.419+0000 7ff5bee5e640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T08:58:47.533 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 08:58:47 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T08:58:47.425+0000 7ff5b6c7d640 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T08:58:48.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:47 smithi114 ceph-mon[33315]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:48.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:47 smithi114 ceph-mon[33315]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:58:48.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:47 smithi114 ceph-mon[33315]: osdmap e48: 8 total, 7 up, 8 in 2024-09-16T08:58:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:47 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:47 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:47 smithi114 ceph-mon[33315]: from='osd.7 ' entity='osd.7' 2024-09-16T08:58:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3170166743' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T08:58:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[29803]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[29803]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:58:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[29803]: osdmap e48: 8 total, 7 up, 8 in 2024-09-16T08:58:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[29803]: from='osd.7 ' entity='osd.7' 2024-09-16T08:58:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3170166743' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T08:58:48.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[40056]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[40056]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-16T08:58:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[40056]: osdmap e48: 8 total, 7 up, 8 in 2024-09-16T08:58:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[40056]: from='osd.7 ' entity='osd.7' 2024-09-16T08:58:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3170166743' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T08:58:48.287 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":48,"num_osds":8,"num_up_osds":7,"osd_up_since":1726477099,"num_in_osds":8,"osd_in_since":1726477105,"num_remapped_pgs":0} 2024-09-16T08:58:49.123 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:48 smithi114 ceph-mon[33315]: purged_snaps scrub starts 2024-09-16T08:58:49.124 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:48 smithi114 ceph-mon[33315]: purged_snaps scrub ok 2024-09-16T08:58:49.124 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:48 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:49.124 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:48 smithi114 ceph-mon[33315]: osd.7 [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] boot 2024-09-16T08:58:49.124 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:48 smithi114 ceph-mon[33315]: osdmap e49: 8 total, 8 up, 8 in 2024-09-16T08:58:49.124 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:48 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:49.124 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:48 smithi114 ceph-mon[33315]: osdmap e50: 8 total, 8 up, 8 in 2024-09-16T08:58:49.288 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T08:58:49.313 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[29803]: purged_snaps scrub starts 2024-09-16T08:58:49.313 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[29803]: purged_snaps scrub ok 2024-09-16T08:58:49.313 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:49.313 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[29803]: osd.7 [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] boot 2024-09-16T08:58:49.313 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[29803]: osdmap e49: 8 total, 8 up, 8 in 2024-09-16T08:58:49.313 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:49.313 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[29803]: osdmap e50: 8 total, 8 up, 8 in 2024-09-16T08:58:49.314 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[40056]: purged_snaps scrub starts 2024-09-16T08:58:49.314 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[40056]: purged_snaps scrub ok 2024-09-16T08:58:49.314 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:49.314 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[40056]: osd.7 [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] boot 2024-09-16T08:58:49.314 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[40056]: osdmap e49: 8 total, 8 up, 8 in 2024-09-16T08:58:49.314 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:58:49.314 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:48 smithi111 ceph-mon[40056]: osdmap e50: 8 total, 8 up, 8 in 2024-09-16T08:58:49.544 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:58:50.036 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 -- 172.21.15.111:0/2787919686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40103440 msgr2=0x7f5a40109f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:50.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 --2- 172.21.15.111:0/2787919686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40103440 0x7f5a40109f90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009cd0 tx=0x7f5a3802a8d0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 -- 172.21.15.111:0/2787919686 shutdown_connections 2024-09-16T08:58:50.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 --2- 172.21.15.111:0/2787919686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40103440 0x7f5a40109f90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 --2- 172.21.15.111:0/2787919686 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a401028d0 0x7f5a40102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 --2- 172.21.15.111:0/2787919686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a40101f90 0x7f5a40102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 -- 172.21.15.111:0/2787919686 >> 172.21.15.111:0/2787919686 conn(0x7f5a400fc970 msgr2=0x7f5a400fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:50.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 -- 172.21.15.111:0/2787919686 shutdown_connections 2024-09-16T08:58:50.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 -- 172.21.15.111:0/2787919686 wait complete. 2024-09-16T08:58:50.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 Processor -- start 2024-09-16T08:58:50.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.027+0000 7f5a47072640 1 -- start start 2024-09-16T08:58:50.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a47072640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a40101f90 0x7f5a40197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:50.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a47072640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a401028d0 0x7f5a40197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:50.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a47072640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a40103440 0x7f5a401984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:50.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a47072640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4010ca40 con 0x7f5a401028d0 2024-09-16T08:58:50.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a47072640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4010c8c0 con 0x7f5a40103440 2024-09-16T08:58:50.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a47072640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f5a4010cbc0 con 0x7f5a40101f90 2024-09-16T08:58:50.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a44e0e640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a40101f90 0x7f5a40197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:50.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a3ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a401028d0 0x7f5a40197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:50.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a44e0e640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a40101f90 0x7f5a40197230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:40912/0 (socket says 172.21.15.111:40912) 2024-09-16T08:58:50.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a44e0e640 1 -- 172.21.15.111:0/1179104445 learned_addr learned my addr 172.21.15.111:0/1179104445 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:58:50.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a4560f640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a40103440 0x7f5a401984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:50.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a44e0e640 1 -- 172.21.15.111:0/1179104445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a40103440 msgr2=0x7f5a401984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:50.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a44e0e640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a40103440 0x7f5a401984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a44e0e640 1 -- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a401028d0 msgr2=0x7f5a40197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:50.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a44e0e640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a401028d0 0x7f5a40197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a44e0e640 1 -- 172.21.15.111:0/1179104445 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a40198d60 con 0x7f5a40101f90 2024-09-16T08:58:50.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a4560f640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a40103440 0x7f5a401984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T08:58:50.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a3ffff640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a401028d0 0x7f5a40197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:58:50.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a44e0e640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a40101f90 0x7f5a40197230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000eb00 tx=0x7f5a30009f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:50.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a3dffb640 1 -- 172.21.15.111:0/1179104445 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3001c070 con 0x7f5a40101f90 2024-09-16T08:58:50.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a3dffb640 1 -- 172.21.15.111:0/1179104445 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f5a3000bc30 con 0x7f5a40101f90 2024-09-16T08:58:50.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.028+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a40199050 con 0x7f5a40101f90 2024-09-16T08:58:50.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.029+0000 7f5a3dffb640 1 -- 172.21.15.111:0/1179104445 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a300163b0 con 0x7f5a40101f90 2024-09-16T08:58:50.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.029+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a40100df0 con 0x7f5a40101f90 2024-09-16T08:58:50.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.030+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a40102390 con 0x7f5a40101f90 2024-09-16T08:58:50.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.030+0000 7f5a3dffb640 1 -- 172.21.15.111:0/1179104445 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f5a3001f070 con 0x7f5a40101f90 2024-09-16T08:58:50.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.031+0000 7f5a3dffb640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5a2c07ad20 0x7f5a2c07d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:50.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.031+0000 7f5a3ffff640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5a2c07ad20 0x7f5a2c07d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:50.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.031+0000 7f5a3dffb640 1 -- 172.21.15.111:0/1179104445 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(51..51 src has 1..51) ==== 5287+0+0 (secure 0 0 0) 0x7f5a3009f100 con 0x7f5a40101f90 2024-09-16T08:58:50.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.031+0000 7f5a3ffff640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5a2c07ad20 0x7f5a2c07d1e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5a280095f0 tx=0x7f5a28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:50.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.035+0000 7f5a3dffb640 1 -- 172.21.15.111:0/1179104445 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f5a3006ac00 con 0x7f5a40101f90 2024-09-16T08:58:50.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.211+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f5a40101a60 con 0x7f5a40101f90 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: Detected new or changed devices on smithi114 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: osdmap e51: 8 total, 8 up, 8 in 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi114 to 4272M 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:50.213 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:50.214 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.214 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:50.214 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: Detected new or changed devices on smithi114 2024-09-16T08:58:50.214 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.214 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.214 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: osdmap e51: 8 total, 8 up, 8 in 2024-09-16T08:58:50.214 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.215 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.215 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.215 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.215 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi114 to 4272M 2024-09-16T08:58:50.215 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.215 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:50.215 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:50.215 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:50 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.212+0000 7f5a3dffb640 1 -- 172.21.15.111:0/1179104445 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v51) ==== 74+0+130 (secure 0 0 0) 0x7f5a3006fab0 con 0x7f5a40101f90 2024-09-16T08:58:50.216 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:58:50.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5a2c07ad20 msgr2=0x7f5a2c07d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:50.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5a2c07ad20 0x7f5a2c07d1e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5a280095f0 tx=0x7f5a28005f90 comp rx=0 tx=0).stop 2024-09-16T08:58:50.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a40101f90 msgr2=0x7f5a40197230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:50.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a40101f90 0x7f5a40197230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000eb00 tx=0x7f5a30009f20 comp rx=0 tx=0).stop 2024-09-16T08:58:50.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 shutdown_connections 2024-09-16T08:58:50.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5a2c07ad20 0x7f5a2c07d1e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a40103440 0x7f5a401984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a401028d0 0x7f5a40197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 --2- 172.21.15.111:0/1179104445 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a40101f90 0x7f5a40197230 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:50.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 >> 172.21.15.111:0/1179104445 conn(0x7f5a400fc970 msgr2=0x7f5a400fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:50.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 shutdown_connections 2024-09-16T08:58:50.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:50.214+0000 7f5a47072640 1 -- 172.21.15.111:0/1179104445 wait complete. 2024-09-16T08:58:50.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: Detected new or changed devices on smithi114 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: osdmap e51: 8 total, 8 up, 8 in 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi114 to 4272M 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:50.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:50 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:50.865 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":51,"num_osds":8,"num_up_osds":8,"osd_up_since":1726477127,"num_in_osds":8,"osd_in_since":1726477105,"num_remapped_pgs":0} 2024-09-16T08:58:50.866 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T08:58:51.157 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:58:51.422 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1179104445' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T08:58:51.422 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1179104445' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T08:58:51.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1179104445' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-16T08:58:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.598+0000 7fd8035ed640 1 -- 172.21.15.111:0/3349024842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7fc107760 msgr2=0x7fd7fc107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.598+0000 7fd8035ed640 1 --2- 172.21.15.111:0/3349024842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7fc107760 0x7fd7fc107bc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009860 tx=0x7fd7ec02a340 comp rx=0 tx=0).stop 2024-09-16T08:58:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.598+0000 7fd8035ed640 1 -- 172.21.15.111:0/3349024842 shutdown_connections 2024-09-16T08:58:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.598+0000 7fd8035ed640 1 --2- 172.21.15.111:0/3349024842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7fc108100 0x7fd7fc10e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.598+0000 7fd8035ed640 1 --2- 172.21.15.111:0/3349024842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7fc107760 0x7fd7fc107bc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.598+0000 7fd8035ed640 1 --2- 172.21.15.111:0/3349024842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7fc102900 0x7fd7fc102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.598+0000 7fd8035ed640 1 -- 172.21.15.111:0/3349024842 >> 172.21.15.111:0/3349024842 conn(0x7fd7fc076f50 msgr2=0x7fd7fc077360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 -- 172.21.15.111:0/3349024842 shutdown_connections 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 -- 172.21.15.111:0/3349024842 wait complete. 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 Processor -- start 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 -- start start 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7fc102900 0x7fd7fc1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7fc107760 0x7fd7fc198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7fc108100 0x7fd7fc19e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:51.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc111150 con 0x7fd7fc107760 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc110fd0 con 0x7fd7fc108100 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd8035ed640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fd7fc1112d0 con 0x7fd7fc102900 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd801389640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7fc102900 0x7fd7fc1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd801389640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7fc102900 0x7fd7fc1988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:40936/0 (socket says 172.21.15.111:40936) 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd800b88640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7fc107760 0x7fd7fc198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.599+0000 7fd801389640 1 -- 172.21.15.111:0/1792995546 learned_addr learned my addr 172.21.15.111:0/1792995546 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd801b8a640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7fc108100 0x7fd7fc19e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd801389640 1 -- 172.21.15.111:0/1792995546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7fc108100 msgr2=0x7fd7fc19e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd801389640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7fc108100 0x7fd7fc19e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd801389640 1 -- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7fc107760 msgr2=0x7fd7fc198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd801389640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7fc107760 0x7fd7fc198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd801389640 1 -- 172.21.15.111:0/1792995546 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7fc19e960 con 0x7fd7fc102900 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd801389640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7fc102900 0x7fd7fc1988b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800e9f0 tx=0x7fd7f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:51.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd7f27fc640 1 -- 172.21.15.111:0/1792995546 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f800cd80 con 0x7fd7fc102900 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7fc19ec50 con 0x7fd7fc102900 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd7f27fc640 1 -- 172.21.15.111:0/1792995546 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fd7f8010880 con 0x7fd7fc102900 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd7f27fc640 1 -- 172.21.15.111:0/1792995546 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8018760 con 0x7fd7fc102900 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.600+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7fc1a5310 con 0x7fd7fc102900 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.601+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7fc102ce0 con 0x7fd7fc102900 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.602+0000 7fd7f27fc640 1 -- 172.21.15.111:0/1792995546 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7fd7f8010430 con 0x7fd7fc102900 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.602+0000 7fd7f27fc640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd7e807ad20 0x7fd7e807d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.602+0000 7fd7f27fc640 1 -- 172.21.15.111:0/1792995546 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(51..51 src has 1..51) ==== 5287+0+0 (secure 0 0 0) 0x7fd7f809de90 con 0x7fd7fc102900 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.602+0000 7fd800b88640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd7e807ad20 0x7fd7e807d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.603+0000 7fd800b88640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd7e807ad20 0x7fd7e807d1e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec02a930 tx=0x7fd7ec0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.607+0000 7fd7f27fc640 1 -- 172.21.15.111:0/1792995546 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fd7f8069a50 con 0x7fd7fc102900 2024-09-16T08:58:51.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.783+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7fc1a5540 con 0x7fd7fc102900 2024-09-16T08:58:51.785 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.784+0000 7fd7f27fc640 1 -- 172.21.15.111:0/1792995546 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+13650 (secure 0 0 0) 0x7fd7f806e900 con 0x7fd7fc102900 2024-09-16T08:58:51.785 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:58:51.786 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":51,"fsid":"f774c998-7408-11ef-bceb-c7b262605968","created":"2024-09-16T08:53:08.313205+0000","modified":"2024-09-16T08:58:49.160754+0000","last_up_change":"2024-09-16T08:58:47.983219+0000","last_in_change":"2024-09-16T08:58:25.803407+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T08:56:27.171375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b1e3493f-2bf8-489d-b890-6cba7976e3c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6803","nonce":1090026984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6805","nonce":1090026984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6809","nonce":1090026984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6807","nonce":1090026984}]},"public_addr":"172.21.15.111:6803/1090026984","cluster_addr":"172.21.15.111:6805/1090026984","heartbeat_back_addr":"172.21.15.111:6809/1090026984","heartbeat_front_addr":"172.21.15.111:6807/1090026984","state":["exists","up"]},{"osd":1,"uuid":"5c855195-4c6d-47c7-9c63-1e1cc3712c1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6811","nonce":3642230577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6813","nonce":3642230577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6817","nonce":3642230577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6815","nonce":3642230577}]},"public_addr":"172.21.15.111:6811/3642230577","cluster_addr":"172.21.15.111:6813/3642230577","heartbeat_back_addr":"172.21.15.111:6817/3642230577","heartbeat_front_addr":"172.21.15.111:6815/3642230577","state":["exists","up"]},{"osd":2,"uuid":"7650515a-da82-4064-83a2-d4cb3665f1ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6819","nonce":3654696929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6821","nonce":3654696929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6825","nonce":3654696929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6823","nonce":3654696929}]},"public_addr":"172.21.15.111:6819/3654696929","cluster_addr":"172.21.15.111:6821/3654696929","heartbeat_back_addr":"172.21.15.111:6825/3654696929","heartbeat_front_addr":"172.21.15.111:6823/3654696929","state":["exists","up"]},{"osd":3,"uuid":"4e778b16-cf6a-405a-8ae3-899c7b7988b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6827","nonce":1538456457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6829","nonce":1538456457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6833","nonce":1538456457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6831","nonce":1538456457}]},"public_addr":"172.21.15.111:6827/1538456457","cluster_addr":"172.21.15.111:6829/1538456457","heartbeat_back_addr":"172.21.15.111:6833/1538456457","heartbeat_front_addr":"172.21.15.111:6831/1538456457","state":["exists","up"]},{"osd":4,"uuid":"2a15db99-3151-4039-81a5-d651e5b5b756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6801","nonce":3634478750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6803","nonce":3634478750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6807","nonce":3634478750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6805","nonce":3634478750}]},"public_addr":"172.21.15.114:6801/3634478750","cluster_addr":"172.21.15.114:6803/3634478750","heartbeat_back_addr":"172.21.15.114:6807/3634478750","heartbeat_front_addr":"172.21.15.114:6805/3634478750","state":["exists","up"]},{"osd":5,"uuid":"45f16b15-1806-477e-96d5-2f84ab4cc399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6809","nonce":101553972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6811","nonce":101553972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6815","nonce":101553972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6813","nonce":101553972}]},"public_addr":"172.21.15.114:6809/101553972","cluster_addr":"172.21.15.114:6811/101553972","heartbeat_back_addr":"172.21.15.114:6815/101553972","heartbeat_front_addr":"172.21.15.114:6813/101553972","state":["exists","up"]},{"osd":6,"uuid":"321fe560-27fd-4b30-a298-ae609af8eec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6817","nonce":3741048002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6819","nonce":3741048002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6823","nonce":3741048002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6821","nonce":3741048002}]},"public_addr":"172.21.15.114:6817/3741048002","cluster_addr":"172.21.15.114:6819/3741048002","heartbeat_back_addr":"172.21.15.114:6823/3741048002","heartbeat_front_addr":"172.21.15.114:6821/3741048002","state":["exists","up"]},{"osd":7,"uuid":"16273109-6547-46b3-b670-6173327b7677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6825","nonce":2543648078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6827","nonce":2543648078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6831","nonce":2543648078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6829","nonce":2543648078}]},"public_addr":"172.21.15.114:6825/2543648078","cluster_addr":"172.21.15.114:6827/2543648078","heartbeat_back_addr":"172.21.15.114:6831/2543648078","heartbeat_front_addr":"172.21.15.114:6829/2543648078","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:55:25.539345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:55:54.372473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:56:24.402273+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:56:53.591048+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:57:20.144327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:57:49.193663+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:58:17.683380+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:58:46.616167+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:0/1868356259":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/2123840083":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/2055672369":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/2883300306":"2024-09-17T08:53:39.765611+0000","172.21.15.111:6801/2366462893":"2024-09-17T08:54:06.869856+0000","172.21.15.111:6801/2026406736":"2024-09-17T08:53:39.765611+0000","172.21.15.111:6800/2026406736":"2024-09-17T08:53:39.765611+0000","172.21.15.111:0/3024901697":"2024-09-17T08:53:39.765611+0000","172.21.15.111:6800/2366462893":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/1333352500":"2024-09-17T08:53:39.765611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T08:58:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd7e807ad20 msgr2=0x7fd7e807d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd7e807ad20 0x7fd7e807d1e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec02a930 tx=0x7fd7ec0058d0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7fc102900 msgr2=0x7fd7fc1988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7fc102900 0x7fd7fc1988b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800e9f0 tx=0x7fd7f800eeb0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 shutdown_connections 2024-09-16T08:58:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd7e807ad20 0x7fd7e807d1e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7fc108100 0x7fd7fc19e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7fc107760 0x7fd7fc198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 --2- 172.21.15.111:0/1792995546 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7fc102900 0x7fd7fc1988b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.786+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 >> 172.21.15.111:0/1792995546 conn(0x7fd7fc076f50 msgr2=0x7fd7fc105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.787+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 shutdown_connections 2024-09-16T08:58:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:51.787+0000 7fd8035ed640 1 -- 172.21.15.111:0/1792995546 wait complete. 2024-09-16T08:58:52.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:52 smithi111 ceph-mon[40056]: pgmap v183: 1 pgs: 1 peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:58:52.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:52 smithi111 ceph-mon[40056]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T08:58:52.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1792995546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T08:58:52.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:52 smithi111 ceph-mon[29803]: pgmap v183: 1 pgs: 1 peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:58:52.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:52 smithi111 ceph-mon[29803]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T08:58:52.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1792995546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T08:58:52.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:52 smithi114 ceph-mon[33315]: pgmap v183: 1 pgs: 1 peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:58:52.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:52 smithi114 ceph-mon[33315]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T08:58:52.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1792995546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T08:58:52.559 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T08:56:27.171375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T08:58:52.559 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-16T08:58:52.804 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:58:53.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.267+0000 7fe6da753640 1 -- 172.21.15.111:0/1956919464 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6d4108030 msgr2=0x7fe6d410e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:53.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.267+0000 7fe6da753640 1 --2- 172.21.15.111:0/1956919464 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6d4108030 0x7fe6d410e830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009ca0 tx=0x7fe6cc02a8c0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.267+0000 7fe6da753640 1 -- 172.21.15.111:0/1956919464 shutdown_connections 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.267+0000 7fe6da753640 1 --2- 172.21.15.111:0/1956919464 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6d4108030 0x7fe6d410e830 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.267+0000 7fe6da753640 1 --2- 172.21.15.111:0/1956919464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d41076b0 0x7fe6d4107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.267+0000 7fe6da753640 1 --2- 172.21.15.111:0/1956919464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d4101ca0 0x7fe6d4102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.267+0000 7fe6da753640 1 -- 172.21.15.111:0/1956919464 >> 172.21.15.111:0/1956919464 conn(0x7fe6d40fd710 msgr2=0x7fe6d40ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.267+0000 7fe6da753640 1 -- 172.21.15.111:0/1956919464 shutdown_connections 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.267+0000 7fe6da753640 1 -- 172.21.15.111:0/1956919464 wait complete. 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6da753640 1 Processor -- start 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6da753640 1 -- start start 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6da753640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6d4101ca0 0x7fe6d4197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6da753640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d41076b0 0x7fe6d4197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:53.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6da753640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d4108030 0x7fe6d419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6da753640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d4111180 con 0x7fe6d4108030 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6da753640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d4111000 con 0x7fe6d41076b0 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6da753640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d4111300 con 0x7fe6d4101ca0 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6d8cf0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d4108030 0x7fe6d419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6d3fff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6d4101ca0 0x7fe6d4197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6d8cf0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d4108030 0x7fe6d419cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40068/0 (socket says 172.21.15.111:40068) 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6d8cf0640 1 -- 172.21.15.111:0/3327307962 learned_addr learned my addr 172.21.15.111:0/3327307962 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.268+0000 7fe6d37fe640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d41076b0 0x7fe6d4197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d3fff640 1 -- 172.21.15.111:0/3327307962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d41076b0 msgr2=0x7fe6d4197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d3fff640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d41076b0 0x7fe6d4197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d3fff640 1 -- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d4108030 msgr2=0x7fe6d419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d3fff640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d4108030 0x7fe6d419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d3fff640 1 -- 172.21.15.111:0/3327307962 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d419d710 con 0x7fe6d4101ca0 2024-09-16T08:58:53.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d8cf0640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d4108030 0x7fe6d419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d37fe640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d41076b0 0x7fe6d4197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d3fff640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6d4101ca0 0x7fe6d4197660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400e9f0 tx=0x7fe6c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d17fa640 1 -- 172.21.15.111:0/3327307962 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c400cd80 con 0x7fe6d4101ca0 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d419da00 con 0x7fe6d4101ca0 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d17fa640 1 -- 172.21.15.111:0/3327307962 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fe6c4004470 con 0x7fe6d4101ca0 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6d17fa640 1 -- 172.21.15.111:0/3327307962 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4003e10 con 0x7fe6d4101ca0 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.269+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d41a4290 con 0x7fe6d4101ca0 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.270+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d4102080 con 0x7fe6d4101ca0 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.271+0000 7fe6d17fa640 1 -- 172.21.15.111:0/3327307962 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7fe6c4003fb0 con 0x7fe6d4101ca0 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.271+0000 7fe6d17fa640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe6bc07ad20 0x7fe6bc07d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:53.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.271+0000 7fe6d17fa640 1 -- 172.21.15.111:0/3327307962 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(51..51 src has 1..51) ==== 5287+0+0 (secure 0 0 0) 0x7fe6c409dd60 con 0x7fe6d4101ca0 2024-09-16T08:58:53.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.271+0000 7fe6d37fe640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe6bc07ad20 0x7fe6bc07d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:53.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.272+0000 7fe6d37fe640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe6bc07ad20 0x7fe6bc07d1e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe6c80058c0 tx=0x7fe6c8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:53.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.276+0000 7fe6d17fa640 1 -- 172.21.15.111:0/3327307962 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fe6c4065480 con 0x7fe6d4101ca0 2024-09-16T08:58:53.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.449+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe6d4109e10 con 0x7fe6d4101ca0 2024-09-16T08:58:53.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.449+0000 7fe6d17fa640 1 -- 172.21.15.111:0/3327307962 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) ==== 93+0+10 (secure 0 0 0) 0x7fe6c4069860 con 0x7fe6d4101ca0 2024-09-16T08:58:53.450 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2024-09-16T08:58:53.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe6bc07ad20 msgr2=0x7fe6bc07d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:53.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe6bc07ad20 0x7fe6bc07d1e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe6c80058c0 tx=0x7fe6c8005850 comp rx=0 tx=0).stop 2024-09-16T08:58:53.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6d4101ca0 msgr2=0x7fe6d4197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:53.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6d4101ca0 0x7fe6d4197660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400e9f0 tx=0x7fe6c400eeb0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 shutdown_connections 2024-09-16T08:58:53.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fe6bc07ad20 0x7fe6bc07d1e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d4108030 0x7fe6d419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d41076b0 0x7fe6d4197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 --2- 172.21.15.111:0/3327307962 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6d4101ca0 0x7fe6d4197660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:53.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 >> 172.21.15.111:0/3327307962 conn(0x7fe6d40fd710 msgr2=0x7fe6d40ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:53.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 shutdown_connections 2024-09-16T08:58:53.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:58:53.452+0000 7fe6da753640 1 -- 172.21.15.111:0/3327307962 wait complete. 2024-09-16T08:58:54.261 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi111 2024-09-16T08:58:54.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi111=foo.a' 2024-09-16T08:58:54.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:54 smithi111 ceph-mon[29803]: pgmap v184: 1 pgs: 1 peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:58:54.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3327307962' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-16T08:58:54.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:54 smithi111 ceph-mon[40056]: pgmap v184: 1 pgs: 1 peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:58:54.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3327307962' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-16T08:58:54.516 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:58:54.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:54 smithi114 ceph-mon[33315]: pgmap v184: 1 pgs: 1 peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:58:54.637 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3327307962' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-16T08:58:54.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 -- 172.21.15.114:0/2168625174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c103400 msgr2=0x7f734c109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:54.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 --2- 172.21.15.114:0/2168625174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c103400 0x7f734c109d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7344009cd0 tx=0x7f734402a8e0 comp rx=0 tx=0).stop 2024-09-16T08:58:54.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 -- 172.21.15.114:0/2168625174 shutdown_connections 2024-09-16T08:58:54.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 --2- 172.21.15.114:0/2168625174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c103400 0x7f734c109d80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:55.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 --2- 172.21.15.114:0/2168625174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734c102a40 0x7f734c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:55.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 --2- 172.21.15.114:0/2168625174 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f734c1018e0 0x7f734c101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:55.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 -- 172.21.15.114:0/2168625174 >> 172.21.15.114:0/2168625174 conn(0x7f734c0fd1b0 msgr2=0x7f734c0ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:55.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 -- 172.21.15.114:0/2168625174 shutdown_connections 2024-09-16T08:58:55.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 -- 172.21.15.114:0/2168625174 wait complete. 2024-09-16T08:58:55.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 Processor -- start 2024-09-16T08:58:55.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.996+0000 7f7352861640 1 -- start start 2024-09-16T08:58:55.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f7352861640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f734c1018e0 0x7f734c06b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:55.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f7352861640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c102a40 0x7f734c19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:55.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f7352861640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734c103400 0x7f734c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:55.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f7352861640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f734c10ca30 con 0x7f734c103400 2024-09-16T08:58:55.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f7352861640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f734c10c8b0 con 0x7f734c102a40 2024-09-16T08:58:55.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f7352861640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f734c10cbb0 con 0x7f734c1018e0 2024-09-16T08:58:55.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c102a40 0x7f734c19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:55.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c102a40 0x7f734c19cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34538/0 (socket says 172.21.15.114:34538) 2024-09-16T08:58:55.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734b7fe640 1 -- 172.21.15.114:0/3885483720 learned_addr learned my addr 172.21.15.114:0/3885483720 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:58:55.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734bfff640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f734c1018e0 0x7f734c06b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:55.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f7350dfe640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734c103400 0x7f734c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:55.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734b7fe640 1 -- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f734c1018e0 msgr2=0x7f734c06b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:55.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734b7fe640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f734c1018e0 0x7f734c06b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:55.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734b7fe640 1 -- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734c103400 msgr2=0x7f734c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:55.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734b7fe640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734c103400 0x7f734c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734b7fe640 1 -- 172.21.15.114:0/3885483720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734c19fea0 con 0x7f734c102a40 2024-09-16T08:58:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f734b7fe640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c102a40 0x7f734c19cff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f734000c890 tx=0x7f734000cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f73497fa640 1 -- 172.21.15.114:0/3885483720 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340004220 con 0x7f734c102a40 2024-09-16T08:58:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f73497fa640 1 -- 172.21.15.114:0/3885483720 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f7340007e00 con 0x7f734c102a40 2024-09-16T08:58:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f73497fa640 1 -- 172.21.15.114:0/3885483720 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73400047a0 con 0x7f734c102a40 2024-09-16T08:58:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f734c1a0190 con 0x7f734c102a40 2024-09-16T08:58:55.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.997+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f734c1a0750 con 0x7f734c102a40 2024-09-16T08:58:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.999+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f734c101ce0 con 0x7f734c102a40 2024-09-16T08:58:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.999+0000 7f73497fa640 1 -- 172.21.15.114:0/3885483720 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f73400079b0 con 0x7f734c102a40 2024-09-16T08:58:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:54.999+0000 7f73497fa640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f733407ad20 0x7f733407d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.000+0000 7f73497fa640 1 -- 172.21.15.114:0/3885483720 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5287+0+0 (secure 0 0 0) 0x7f734009da00 con 0x7f734c102a40 2024-09-16T08:58:55.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.000+0000 7f734bfff640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f733407ad20 0x7f733407d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:55.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.000+0000 7f734bfff640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f733407ad20 0x7f733407d1e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f733c009990 tx=0x7f733c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:55.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.004+0000 7f73497fa640 1 -- 172.21.15.114:0/3885483720 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f7340014030 con 0x7f734c102a40 2024-09-16T08:58:55.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.191+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi111=foo.a", "target": ["mon-mgr", ""]}) -- 0x7f734c063590 con 0x7f733407ad20 2024-09-16T08:58:55.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.201+0000 7f73497fa640 1 -- 172.21.15.114:0/3885483720 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f734c063590 con 0x7f733407ad20 2024-09-16T08:58:55.201 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled rgw.foo.a update... 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f733407ad20 msgr2=0x7f733407d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f733407ad20 0x7f733407d1e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f733c009990 tx=0x7f733c009110 comp rx=0 tx=0).stop 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c102a40 msgr2=0x7f734c19cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c102a40 0x7f734c19cff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f734000c890 tx=0x7f734000cc90 comp rx=0 tx=0).stop 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 shutdown_connections 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f733407ad20 0x7f733407d1e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734c103400 0x7f734c19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f734c102a40 0x7f734c19cff0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 --2- 172.21.15.114:0/3885483720 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f734c1018e0 0x7f734c06b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 >> 172.21.15.114:0/3885483720 conn(0x7f734c0fd1b0 msgr2=0x7f734c0fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 shutdown_connections 2024-09-16T08:58:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:55.203+0000 7f7352861640 1 -- 172.21.15.114:0/3885483720 wait complete. 2024-09-16T08:58:55.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:55 smithi114 ceph-mon[33315]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T08:58:55.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:55 smithi114 ceph-mon[33315]: Cluster is now healthy 2024-09-16T08:58:55.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:55.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:55.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:55.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:55.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:55.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-16T08:58:55.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:55 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-16T08:58:55.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[29803]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T08:58:55.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[29803]: Cluster is now healthy 2024-09-16T08:58:55.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:55.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:55.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:55.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:55.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[40056]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[40056]: Cluster is now healthy 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-16T08:58:55.608 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:55 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-16T08:58:55.808 DEBUG:teuthology.orchestra.run.smithi111:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@rgw.foo.a.service 2024-09-16T08:58:55.811 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi114 2024-09-16T08:58:55.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-16T08:58:56.053 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:58:56.458 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:56 smithi114 ceph-mon[33315]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-16T08:58:56.458 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:56 smithi114 ceph-mon[33315]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi111=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:58:56.459 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:56 smithi114 ceph-mon[33315]: Saving service rgw.foo.a spec with placement smithi111=foo.a;count:1 2024-09-16T08:58:56.459 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:56.459 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:56 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:56.459 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:56 smithi114 ceph-mon[33315]: Deploying daemon rgw.foo.a on smithi111 2024-09-16T08:58:56.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.456+0000 7fb0f7017640 1 -- 172.21.15.114:0/1486906076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f0105be0 msgr2=0x7fb0f0107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:56.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.456+0000 7fb0f7017640 1 --2- 172.21.15.114:0/1486906076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f0105be0 0x7fb0f0107ff0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009900 tx=0x7fb0ec02a410 comp rx=0 tx=0).stop 2024-09-16T08:58:56.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.456+0000 7fb0f7017640 1 -- 172.21.15.114:0/1486906076 shutdown_connections 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.456+0000 7fb0f7017640 1 --2- 172.21.15.114:0/1486906076 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0108730 0x7fb0f010abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.456+0000 7fb0f7017640 1 --2- 172.21.15.114:0/1486906076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f0105be0 0x7fb0f0107ff0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.456+0000 7fb0f7017640 1 --2- 172.21.15.114:0/1486906076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f006be30 0x7fb0f01056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.456+0000 7fb0f7017640 1 -- 172.21.15.114:0/1486906076 >> 172.21.15.114:0/1486906076 conn(0x7fb0f00fd210 msgr2=0x7fb0f00ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 -- 172.21.15.114:0/1486906076 shutdown_connections 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 -- 172.21.15.114:0/1486906076 wait complete. 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 Processor -- start 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 -- start start 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f006be30 0x7fb0f01982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:56.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f0105be0 0x7fb0f0198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0108730 0x7fb0f0199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f010daf0 con 0x7fb0f0105be0 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f010d970 con 0x7fb0f006be30 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f7017640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fb0f010dc70 con 0x7fb0f0108730 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f4db3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f006be30 0x7fb0f01982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f4db3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f006be30 0x7fb0f01982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34568/0 (socket says 172.21.15.114:34568) 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.457+0000 7fb0f4db3640 1 -- 172.21.15.114:0/158225180 learned_addr learned my addr 172.21.15.114:0/158225180 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0e7fff640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f0105be0 0x7fb0f0198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0f55b4640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0108730 0x7fb0f0199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0f4db3640 1 -- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0108730 msgr2=0x7fb0f0199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0f4db3640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0108730 0x7fb0f0199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0f4db3640 1 -- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f0105be0 msgr2=0x7fb0f0198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0f4db3640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f0105be0 0x7fb0f0198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0f4db3640 1 -- 172.21.15.114:0/158225180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f0199df0 con 0x7fb0f006be30 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0f4db3640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f006be30 0x7fb0f01982c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000d950 tx=0x7fb0e000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0e5ffb640 1 -- 172.21.15.114:0/158225180 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0017070 con 0x7fb0f006be30 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0e5ffb640 1 -- 172.21.15.114:0/158225180 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fb0e0009c30 con 0x7fb0f006be30 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f019a0e0 con 0x7fb0f006be30 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0e5ffb640 1 -- 172.21.15.114:0/158225180 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e000f3c0 con 0x7fb0f006be30 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.458+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f01a07c0 con 0x7fb0f006be30 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.459+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0f0105fe0 con 0x7fb0f006be30 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.460+0000 7fb0e5ffb640 1 -- 172.21.15.114:0/158225180 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7fb0e0010460 con 0x7fb0f006be30 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.460+0000 7fb0e5ffb640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d807ad20 0x7fb0d807d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:58:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.461+0000 7fb0e5ffb640 1 -- 172.21.15.114:0/158225180 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5287+0+0 (secure 0 0 0) 0x7fb0e009d620 con 0x7fb0f006be30 2024-09-16T08:58:56.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.461+0000 7fb0e7fff640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d807ad20 0x7fb0d807d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:58:56.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.461+0000 7fb0e7fff640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d807ad20 0x7fb0d807d1e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec000c00 tx=0x7fb0ec005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:58:56.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.466+0000 7fb0e5ffb640 1 -- 172.21.15.114:0/158225180 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fb0e00691a0 con 0x7fb0f006be30 2024-09-16T08:58:56.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:56.641+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) -- 0x7fb0f002e830 con 0x7fb0f006be30 2024-09-16T08:58:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[29803]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-16T08:58:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[29803]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi111=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:58:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[29803]: Saving service rgw.foo.a spec with placement smithi111=foo.a;count:1 2024-09-16T08:58:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[29803]: Deploying daemon rgw.foo.a on smithi111 2024-09-16T08:58:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[40056]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-16T08:58:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[40056]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi111=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:58:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[40056]: Saving service rgw.foo.a spec with placement smithi111=foo.a;count:1 2024-09-16T08:58:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:56 smithi111 ceph-mon[40056]: Deploying daemon rgw.foo.a on smithi111 2024-09-16T08:58:57.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.240+0000 7fb0e5ffb640 1 -- 172.21.15.114:0/158225180 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v52) ==== 160+0+0 (secure 0 0 0) 0x7fb0e006e050 con 0x7fb0f006be30 2024-09-16T08:58:57.242 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'datapool' created 2024-09-16T08:58:57.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.242+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d807ad20 msgr2=0x7fb0d807d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:57.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.242+0000 7fb0f7017640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d807ad20 0x7fb0d807d1e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec000c00 tx=0x7fb0ec005e20 comp rx=0 tx=0).stop 2024-09-16T08:58:57.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.242+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f006be30 msgr2=0x7fb0f01982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:58:57.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.242+0000 7fb0f7017640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f006be30 0x7fb0f01982c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000d950 tx=0x7fb0e000de10 comp rx=0 tx=0).stop 2024-09-16T08:58:57.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.242+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 shutdown_connections 2024-09-16T08:58:57.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.242+0000 7fb0f7017640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d807ad20 0x7fb0d807d1e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:57.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.242+0000 7fb0f7017640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0108730 0x7fb0f0199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:57.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.242+0000 7fb0f7017640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f0105be0 0x7fb0f0198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:57.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.242+0000 7fb0f7017640 1 --2- 172.21.15.114:0/158225180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f006be30 0x7fb0f01982c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:58:57.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.243+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 >> 172.21.15.114:0/158225180 conn(0x7fb0f00fd210 msgr2=0x7fb0f0103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:58:57.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.243+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 shutdown_connections 2024-09-16T08:58:57.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:58:57.243+0000 7fb0f7017640 1 -- 172.21.15.114:0/158225180 wait complete. 2024-09-16T08:58:57.391 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/158225180' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T08:58:57.392 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T08:58:57.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/158225180' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T08:58:57.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T08:58:57.619 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/158225180' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T08:58:57.619 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-16T08:58:57.661 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 08:58:57 smithi111 podman[66875]: 2024-09-16 08:58:57.416328677 +0000 UTC m=+0.138386701 container create ce5ec9cb79b078018b78a0d4ec2fb7e4afd11f763008f33694cc2e84db463e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-16T08:58:57.661 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 08:58:57 smithi111 podman[66875]: 2024-09-16 08:58:57.660841884 +0000 UTC m=+0.382899908 container init ce5ec9cb79b078018b78a0d4ec2fb7e4afd11f763008f33694cc2e84db463e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-16T08:58:57.926 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 08:58:57 smithi111 podman[66875]: 2024-09-16 08:58:57.665903247 +0000 UTC m=+0.387961271 container start ce5ec9cb79b078018b78a0d4ec2fb7e4afd11f763008f33694cc2e84db463e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-16T08:58:57.926 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 08:58:57 smithi111 bash[66875]: ce5ec9cb79b078018b78a0d4ec2fb7e4afd11f763008f33694cc2e84db463e05 2024-09-16T08:58:57.926 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 08:58:57 smithi111 systemd[1]: Started Ceph rgw.foo.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:58:57.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- rbd pool init datapool 2024-09-16T08:58:58.182 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:58:58.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-16T08:58:58.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-16T08:58:58.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: osdmap e52: 8 total, 8 up, 8 in 2024-09-16T08:58:58.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: Saving service rgw.foo.a spec with placement smithi111=foo.a;count:1 2024-09-16T08:58:58.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: osdmap e52: 8 total, 8 up, 8 in 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: Saving service rgw.foo.a spec with placement smithi111=foo.a;count:1 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.554 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:58 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: osdmap e52: 8 total, 8 up, 8 in 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: Saving service rgw.foo.a spec with placement smithi111=foo.a;count:1 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:58.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:58 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:58:59.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:59 smithi114 ceph-mon[33315]: osdmap e53: 8 total, 8 up, 8 in 2024-09-16T08:58:59.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/44600549' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-16T08:58:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/3109725613' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T08:58:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T08:58:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:59 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:59 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:59 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:58:59 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[40056]: osdmap e53: 8 total, 8 up, 8 in 2024-09-16T08:58:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/44600549' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-16T08:58:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/3109725613' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T08:58:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T08:58:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:58:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[29803]: osdmap e53: 8 total, 8 up, 8 in 2024-09-16T08:58:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/44600549' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-16T08:58:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/3109725613' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T08:58:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-16T08:58:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:58:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:58:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:58:59 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:59:00.601 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[29803]: pgmap v189: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[29803]: Checking dashboard <-> RGW credentials 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[29803]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/44600549' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[29803]: osdmap e54: 8 total, 8 up, 8 in 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[40056]: pgmap v189: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[40056]: Checking dashboard <-> RGW credentials 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[40056]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/44600549' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-16T08:59:00.602 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-16T08:59:00.603 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:00 smithi111 ceph-mon[40056]: osdmap e54: 8 total, 8 up, 8 in 2024-09-16T08:59:00.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:00 smithi114 ceph-mon[33315]: pgmap v189: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T08:59:00.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:00 smithi114 ceph-mon[33315]: Checking dashboard <-> RGW credentials 2024-09-16T08:59:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:00 smithi114 ceph-mon[33315]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T08:59:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/44600549' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-16T08:59:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-16T08:59:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:00 smithi114 ceph-mon[33315]: osdmap e54: 8 total, 8 up, 8 in 2024-09-16T08:59:01.618 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:01 smithi111 ceph-mon[29803]: osdmap e55: 8 total, 8 up, 8 in 2024-09-16T08:59:01.618 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T08:59:01.618 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T08:59:01.618 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:01 smithi111 ceph-mon[40056]: osdmap e55: 8 total, 8 up, 8 in 2024-09-16T08:59:01.618 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T08:59:01.619 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T08:59:01.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:01 smithi114 ceph-mon[33315]: osdmap e55: 8 total, 8 up, 8 in 2024-09-16T08:59:01.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T08:59:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-16T08:59:01.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.114 --placement '1;smithi114=iscsi.a' 2024-09-16T08:59:02.155 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:59:02.526 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:02 smithi114 ceph-mon[33315]: pgmap v192: 68 pgs: 8 creating+peering, 10 active+clean, 50 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-16T08:59:02.527 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T08:59:02.527 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T08:59:02.527 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:02 smithi114 ceph-mon[33315]: osdmap e56: 8 total, 8 up, 8 in 2024-09-16T08:59:02.527 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:02 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.611+0000 7f589d305640 1 -- 172.21.15.114:0/679895660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58981029f0 msgr2=0x7f5898102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.611+0000 7f589d305640 1 --2- 172.21.15.114:0/679895660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58981029f0 0x7f5898102df0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5888009930 tx=0x7f588802a420 comp rx=0 tx=0).stop 2024-09-16T08:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.611+0000 7f589d305640 1 -- 172.21.15.114:0/679895660 shutdown_connections 2024-09-16T08:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.611+0000 7f589d305640 1 --2- 172.21.15.114:0/679895660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5898104510 0x7f589810ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.611+0000 7f589d305640 1 --2- 172.21.15.114:0/679895660 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5898103b50 0x7f5898103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.611+0000 7f589d305640 1 --2- 172.21.15.114:0/679895660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58981029f0 0x7f5898102df0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.611+0000 7f589d305640 1 -- 172.21.15.114:0/679895660 >> 172.21.15.114:0/679895660 conn(0x7f58980fe2c0 msgr2=0x7f5898100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.611+0000 7f589d305640 1 -- 172.21.15.114:0/679895660 shutdown_connections 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.611+0000 7f589d305640 1 -- 172.21.15.114:0/679895660 wait complete. 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f589d305640 1 Processor -- start 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f589d305640 1 -- start start 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f589d305640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58981029f0 0x7f5898198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f589d305640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5898103b50 0x7f58981989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f589d305640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5898104510 0x7f5898199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f589d305640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f589810dae0 con 0x7f5898103b50 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f589d305640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f589810d960 con 0x7f5898104510 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f589d305640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f589810dc60 con 0x7f58981029f0 2024-09-16T08:59:02.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f58977fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5898104510 0x7f5898199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f58977fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5898104510 0x7f5898199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39116/0 (socket says 172.21.15.114:39116) 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f5896ffd640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58981029f0 0x7f5898198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f58977fe640 1 -- 172.21.15.114:0/1454386615 learned_addr learned my addr 172.21.15.114:0/1454386615 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.612+0000 7f58967fc640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5898103b50 0x7f58981989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f58977fe640 1 -- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58981029f0 msgr2=0x7f5898198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f58977fe640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58981029f0 0x7f5898198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f58977fe640 1 -- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5898103b50 msgr2=0x7f58981989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f58977fe640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5898103b50 0x7f58981989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f58977fe640 1 -- 172.21.15.114:0/1454386615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5898199ec0 con 0x7f5898104510 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f58977fe640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5898104510 0x7f5898199620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f589000eb10 tx=0x7f589000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f587ffff640 1 -- 172.21.15.114:0/1454386615 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890004b20 con 0x7f5898104510 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f587ffff640 1 -- 172.21.15.114:0/1454386615 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f5890004cc0 con 0x7f5898104510 2024-09-16T08:59:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f589819a1b0 con 0x7f5898104510 2024-09-16T08:59:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f587ffff640 1 -- 172.21.15.114:0/1454386615 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890010430 con 0x7f5898104510 2024-09-16T08:59:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.613+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58981a0890 con 0x7f5898104510 2024-09-16T08:59:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.614+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5898102df0 con 0x7f5898104510 2024-09-16T08:59:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.614+0000 7f587ffff640 1 -- 172.21.15.114:0/1454386615 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f5890005020 con 0x7f5898104510 2024-09-16T08:59:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.615+0000 7f587ffff640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f588007ad20 0x7f588007d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.615+0000 7f587ffff640 1 -- 172.21.15.114:0/1454386615 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6755+0+0 (secure 0 0 0) 0x7f589009e5e0 con 0x7f5898104510 2024-09-16T08:59:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.615+0000 7f5896ffd640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f588007ad20 0x7f588007d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.615+0000 7f5896ffd640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f588007ad20 0x7f588007d1e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f58880057c0 tx=0x7f58880058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:02.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.619+0000 7f587ffff640 1 -- 172.21.15.114:0/1454386615 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f5890005310 con 0x7f5898104510 2024-09-16T08:59:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[29803]: pgmap v192: 68 pgs: 8 creating+peering, 10 active+clean, 50 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-16T08:59:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T08:59:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T08:59:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[29803]: osdmap e56: 8 total, 8 up, 8 in 2024-09-16T08:59:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[40056]: pgmap v192: 68 pgs: 8 creating+peering, 10 active+clean, 50 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-16T08:59:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T08:59:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-16T08:59:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[40056]: osdmap e56: 8 total, 8 up, 8 in 2024-09-16T08:59:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:02 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:02.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.811+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.114", "placement": "1;smithi114=iscsi.a", "target": ["mon-mgr", ""]}) -- 0x7f589802e8c0 con 0x7f588007ad20 2024-09-16T08:59:02.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.818+0000 7f587ffff640 1 -- 172.21.15.114:0/1454386615 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+35 (secure 0 0 0) 0x7f589802e8c0 con 0x7f588007ad20 2024-09-16T08:59:02.820 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled iscsi.datapool update... 2024-09-16T08:59:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f588007ad20 msgr2=0x7f588007d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f588007ad20 0x7f588007d1e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f58880057c0 tx=0x7f58880058d0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5898104510 msgr2=0x7f5898199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5898104510 0x7f5898199620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f589000eb10 tx=0x7f589000c2d0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 shutdown_connections 2024-09-16T08:59:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f588007ad20 0x7f588007d1e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5898104510 0x7f5898199620 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5898103b50 0x7f58981989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 --2- 172.21.15.114:0/1454386615 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58981029f0 0x7f5898198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 >> 172.21.15.114:0/1454386615 conn(0x7f58980fe2c0 msgr2=0x7f5898100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 shutdown_connections 2024-09-16T08:59:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:02.820+0000 7f589d305640 1 -- 172.21.15.114:0/1454386615 wait complete. 2024-09-16T08:59:03.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:03 smithi114 ceph-mon[33315]: osdmap e57: 8 total, 8 up, 8 in 2024-09-16T08:59:03.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T08:59:03.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T08:59:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:03 smithi114 ceph-mon[33315]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.114", "placement": "1;smithi114=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:03 smithi114 ceph-mon[33315]: Saving service iscsi.datapool spec with placement smithi114=iscsi.a;count:1 2024-09-16T08:59:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:03 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T08:59:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T08:59:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:03 smithi114 ceph-mon[33315]: osdmap e58: 8 total, 8 up, 8 in 2024-09-16T08:59:03.459 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-16T08:59:03.459 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:59:03.459 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-16T08:59:03.488 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:59:03.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-16T08:59:03.514 DEBUG:teuthology.orchestra.run.smithi114:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@iscsi.iscsi.a.service 2024-09-16T08:59:03.557 INFO:tasks.cephadm:Adding prometheus.a on smithi114 2024-09-16T08:59:03.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi114=a' 2024-09-16T08:59:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[40056]: osdmap e57: 8 total, 8 up, 8 in 2024-09-16T08:59:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T08:59:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T08:59:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[40056]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.114", "placement": "1;smithi114=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[40056]: Saving service iscsi.datapool spec with placement smithi114=iscsi.a;count:1 2024-09-16T08:59:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T08:59:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T08:59:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[40056]: osdmap e58: 8 total, 8 up, 8 in 2024-09-16T08:59:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[29803]: osdmap e57: 8 total, 8 up, 8 in 2024-09-16T08:59:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T08:59:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-16T08:59:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[29803]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.114", "placement": "1;smithi114=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[29803]: Saving service iscsi.datapool spec with placement smithi114=iscsi.a;count:1 2024-09-16T08:59:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T08:59:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-16T08:59:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:03 smithi111 ceph-mon[29803]: osdmap e58: 8 total, 8 up, 8 in 2024-09-16T08:59:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:59:04.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.268+0000 7f8672a4a640 1 -- 172.21.15.114:0/3622730631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f866c103b60 msgr2=0x7f866c103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:04.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.268+0000 7f8672a4a640 1 --2- 172.21.15.114:0/3622730631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f866c103b60 0x7f866c103fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8654009930 tx=0x7f865402a420 comp rx=0 tx=0).stop 2024-09-16T08:59:04.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.268+0000 7f8672a4a640 1 -- 172.21.15.114:0/3622730631 shutdown_connections 2024-09-16T08:59:04.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.268+0000 7f8672a4a640 1 --2- 172.21.15.114:0/3622730631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f866c104520 0x7f866c10aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.268+0000 7f8672a4a640 1 --2- 172.21.15.114:0/3622730631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f866c103b60 0x7f866c103fe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.268+0000 7f8672a4a640 1 --2- 172.21.15.114:0/3622730631 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f866c102a00 0x7f866c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.268+0000 7f8672a4a640 1 -- 172.21.15.114:0/3622730631 >> 172.21.15.114:0/3622730631 conn(0x7f866c0fe2f0 msgr2=0x7f866c100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:04.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.268+0000 7f8672a4a640 1 -- 172.21.15.114:0/3622730631 shutdown_connections 2024-09-16T08:59:04.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.268+0000 7f8672a4a640 1 -- 172.21.15.114:0/3622730631 wait complete. 2024-09-16T08:59:04.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.269+0000 7f8672a4a640 1 Processor -- start 2024-09-16T08:59:04.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.269+0000 7f8672a4a640 1 -- start start 2024-09-16T08:59:04.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.269+0000 7f8672a4a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f866c102a00 0x7f866c076cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:04.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.270+0000 7f8672a4a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f866c103b60 0x7f866c075540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:04.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.270+0000 7f8672a4a640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f866c104520 0x7f866c075a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:04.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.270+0000 7f8672a4a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f866c10daf0 con 0x7f866c103b60 2024-09-16T08:59:04.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.270+0000 7f8672a4a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f866c10d970 con 0x7f866c102a00 2024-09-16T08:59:04.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.270+0000 7f8672a4a640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f866c10dc70 con 0x7f866c104520 2024-09-16T08:59:04.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.270+0000 7f866bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f866c102a00 0x7f866c076cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.270+0000 7f8663fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f866c103b60 0x7f866c075540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.270+0000 7f8663fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f866c103b60 0x7f866c075540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.114:56118/0 (socket says 172.21.15.114:56118) 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.270+0000 7f8663fff640 1 -- 172.21.15.114:0/2949211822 learned_addr learned my addr 172.21.15.114:0/2949211822 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.272+0000 7f866bfff640 1 -- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f866c104520 msgr2=0x7f866c075a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.272+0000 7f866bfff640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f866c104520 0x7f866c075a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.272+0000 7f866bfff640 1 -- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f866c103b60 msgr2=0x7f866c075540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.272+0000 7f866bfff640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f866c103b60 0x7f866c075540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.272+0000 7f866bfff640 1 -- 172.21.15.114:0/2949211822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866c076340 con 0x7f866c102a00 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.272+0000 7f8663fff640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f866c103b60 0x7f866c075540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:04.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.272+0000 7f866bfff640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f866c102a00 0x7f866c076cd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f865c00c970 tx=0x7f865c00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.274+0000 7f8669ffb640 1 -- 172.21.15.114:0/2949211822 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c007bb0 con 0x7f866c102a00 2024-09-16T08:59:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.274+0000 7f8669ffb640 1 -- 172.21.15.114:0/2949211822 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f865c007d50 con 0x7f866c102a00 2024-09-16T08:59:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.274+0000 7f8669ffb640 1 -- 172.21.15.114:0/2949211822 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c018660 con 0x7f866c102a00 2024-09-16T08:59:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.275+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866c1a23c0 con 0x7f866c102a00 2024-09-16T08:59:04.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.275+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866c1a2880 con 0x7f866c102a00 2024-09-16T08:59:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.278+0000 7f8669ffb640 1 -- 172.21.15.114:0/2949211822 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f865c005020 con 0x7f866c102a00 2024-09-16T08:59:04.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.278+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f864c0050b0 con 0x7f866c102a00 2024-09-16T08:59:04.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.279+0000 7f8669ffb640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f865807ad70 0x7f865807d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:04.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.279+0000 7f8669ffb640 1 -- 172.21.15.114:0/2949211822 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 7121+0+0 (secure 0 0 0) 0x7f865c09e080 con 0x7f866c102a00 2024-09-16T08:59:04.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.279+0000 7f8663fff640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f865807ad70 0x7f865807d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:04.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.280+0000 7f8663fff640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f865807ad70 0x7f865807d230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f86540096c0 tx=0x7f86540058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:04.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.284+0000 7f8669ffb640 1 -- 172.21.15.114:0/2949211822 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f865c014030 con 0x7f866c102a00 2024-09-16T08:59:04.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.476+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}) -- 0x7f864c000bf0 con 0x7f865807ad70 2024-09-16T08:59:04.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.483+0000 7f8669ffb640 1 -- 172.21.15.114:0/2949211822 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+31 (secure 0 0 0) 0x7f864c000bf0 con 0x7f865807ad70 2024-09-16T08:59:04.485 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled prometheus update... 2024-09-16T08:59:04.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.485+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f865807ad70 msgr2=0x7f865807d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.485+0000 7f8672a4a640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f865807ad70 0x7f865807d230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f86540096c0 tx=0x7f86540058d0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.485+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f866c102a00 msgr2=0x7f866c076cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.486+0000 7f8672a4a640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f866c102a00 0x7f866c076cd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f865c00c970 tx=0x7f865c00ce30 comp rx=0 tx=0).stop 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.486+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 shutdown_connections 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.486+0000 7f8672a4a640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f865807ad70 0x7f865807d230 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.486+0000 7f8672a4a640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f866c104520 0x7f866c075a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.486+0000 7f8672a4a640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f866c103b60 0x7f866c075540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.486+0000 7f8672a4a640 1 --2- 172.21.15.114:0/2949211822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f866c102a00 0x7f866c076cd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.486+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 >> 172.21.15.114:0/2949211822 conn(0x7f866c0fe2f0 msgr2=0x7f866c0fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.486+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 shutdown_connections 2024-09-16T08:59:04.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:04.486+0000 7f8672a4a640 1 -- 172.21.15.114:0/2949211822 wait complete. 2024-09-16T08:59:05.065 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:04 smithi114 ceph-mon[33315]: pgmap v195: 100 pgs: 11 creating+peering, 57 active+clean, 32 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-16T08:59:05.065 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:04 smithi114 ceph-mon[33315]: osdmap e59: 8 total, 8 up, 8 in 2024-09-16T08:59:05.065 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T08:59:05.065 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T08:59:05.065 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:04 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:05.092 DEBUG:teuthology.orchestra.run.smithi114:prometheus.a> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-16T08:59:05.094 INFO:tasks.cephadm:Adding node-exporter.a on smithi111 2024-09-16T08:59:05.094 INFO:tasks.cephadm:Adding node-exporter.b on smithi114 2024-09-16T08:59:05.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi111=a;smithi114=b' 2024-09-16T08:59:05.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[29803]: pgmap v195: 100 pgs: 11 creating+peering, 57 active+clean, 32 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-16T08:59:05.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[29803]: osdmap e59: 8 total, 8 up, 8 in 2024-09-16T08:59:05.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T08:59:05.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T08:59:05.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:05.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[40056]: pgmap v195: 100 pgs: 11 creating+peering, 57 active+clean, 32 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-16T08:59:05.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[40056]: osdmap e59: 8 total, 8 up, 8 in 2024-09-16T08:59:05.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T08:59:05.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-16T08:59:05.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:04 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:05.384 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:59:05.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.815+0000 7f7b088a0640 1 -- 172.21.15.114:0/1305586075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b04105ec0 msgr2=0x7f7b041082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.815+0000 7f7b088a0640 1 --2- 172.21.15.114:0/1305586075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b04105ec0 0x7f7b041082b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009900 tx=0x7f7af802a410 comp rx=0 tx=0).stop 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.815+0000 7f7b088a0640 1 -- 172.21.15.114:0/1305586075 shutdown_connections 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.815+0000 7f7b088a0640 1 --2- 172.21.15.114:0/1305586075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b041087f0 0x7f7b0410abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.815+0000 7f7b088a0640 1 --2- 172.21.15.114:0/1305586075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b04105ec0 0x7f7b041082b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.815+0000 7f7b088a0640 1 --2- 172.21.15.114:0/1305586075 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b04100df0 0x7f7b041011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.815+0000 7f7b088a0640 1 -- 172.21.15.114:0/1305586075 >> 172.21.15.114:0/1305586075 conn(0x7f7b040fc960 msgr2=0x7f7b040feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.815+0000 7f7b088a0640 1 -- 172.21.15.114:0/1305586075 shutdown_connections 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.815+0000 7f7b088a0640 1 -- 172.21.15.114:0/1305586075 wait complete. 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b088a0640 1 Processor -- start 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b088a0640 1 -- start start 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b088a0640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b04100df0 0x7f7b04198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:05.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b088a0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b04105ec0 0x7f7b04198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b088a0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b041087f0 0x7f7b041995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b088a0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0410dae0 con 0x7f7b04105ec0 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b088a0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0410d960 con 0x7f7b041087f0 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b088a0640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0410dc60 con 0x7f7b04100df0 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b02d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b041087f0 0x7f7b041995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b02d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b041087f0 0x7f7b041995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39148/0 (socket says 172.21.15.114:39148) 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b02d9d640 1 -- 172.21.15.114:0/2883025091 learned_addr learned my addr 172.21.15.114:0/2883025091 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b0259c640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b04100df0 0x7f7b04198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b01d9b640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b04105ec0 0x7f7b04198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b02d9d640 1 -- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b04100df0 msgr2=0x7f7b04198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b02d9d640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b04100df0 0x7f7b04198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b02d9d640 1 -- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b04105ec0 msgr2=0x7f7b04198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b02d9d640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b04105ec0 0x7f7b04198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:05.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.816+0000 7f7b02d9d640 1 -- 172.21.15.114:0/2883025091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b04199e50 con 0x7f7b041087f0 2024-09-16T08:59:05.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.817+0000 7f7b02d9d640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b041087f0 0x7f7b041995b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00eaa0 tx=0x7f7afc00ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:05.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.817+0000 7f7af37fe640 1 -- 172.21.15.114:0/2883025091 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc00cdb0 con 0x7f7b041087f0 2024-09-16T08:59:05.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.817+0000 7f7af37fe640 1 -- 172.21.15.114:0/2883025091 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f7afc005020 con 0x7f7b041087f0 2024-09-16T08:59:05.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.817+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0419a140 con 0x7f7b041087f0 2024-09-16T08:59:05.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.817+0000 7f7af37fe640 1 -- 172.21.15.114:0/2883025091 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc0116d0 con 0x7f7b041087f0 2024-09-16T08:59:05.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.817+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b041a0820 con 0x7f7b041087f0 2024-09-16T08:59:05.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.818+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b041011f0 con 0x7f7b041087f0 2024-09-16T08:59:05.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.819+0000 7f7af37fe640 1 -- 172.21.15.114:0/2883025091 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f7afc004a40 con 0x7f7b041087f0 2024-09-16T08:59:05.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.819+0000 7f7af37fe640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7aec07ad20 0x7f7aec07d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:05.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.819+0000 7f7af37fe640 1 -- 172.21.15.114:0/2883025091 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 7132+0+0 (secure 0 0 0) 0x7f7afc09e3c0 con 0x7f7b041087f0 2024-09-16T08:59:05.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.819+0000 7f7b0259c640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7aec07ad20 0x7f7aec07d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:05.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.820+0000 7f7b0259c640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7aec07ad20 0x7f7aec07d1e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7af4005fd0 tx=0x7f7af400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:05.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:05.823+0000 7f7af37fe640 1 -- 172.21.15.114:0/2883025091 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f7afc016030 con 0x7f7b041087f0 2024-09-16T08:59:06.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.015+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi111=a;smithi114=b", "target": ["mon-mgr", ""]}) -- 0x7f7b04107ad0 con 0x7f7aec07ad20 2024-09-16T08:59:06.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.022+0000 7f7af37fe640 1 -- 172.21.15.114:0/2883025091 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+34 (secure 0 0 0) 0x7f7b04107ad0 con 0x7f7aec07ad20 2024-09-16T08:59:06.024 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled node-exporter update... 2024-09-16T08:59:06.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7aec07ad20 msgr2=0x7f7aec07d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7aec07ad20 0x7f7aec07d1e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7af4005fd0 tx=0x7f7af400b390 comp rx=0 tx=0).stop 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b041087f0 msgr2=0x7f7b041995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b041087f0 0x7f7b041995b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00eaa0 tx=0x7f7afc00ef60 comp rx=0 tx=0).stop 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 shutdown_connections 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7aec07ad20 0x7f7aec07d1e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b041087f0 0x7f7b041995b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b04105ec0 0x7f7b04198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 --2- 172.21.15.114:0/2883025091 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b04100df0 0x7f7b04198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.024+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 >> 172.21.15.114:0/2883025091 conn(0x7f7b040fc960 msgr2=0x7f7b04105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.025+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 shutdown_connections 2024-09-16T08:59:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:06.025+0000 7f7b088a0640 1 -- 172.21.15.114:0/2883025091 wait complete. 2024-09-16T08:59:06.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:05 smithi114 ceph-mon[33315]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:06.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:05 smithi114 ceph-mon[33315]: Saving service prometheus spec with placement smithi114=a;count:1 2024-09-16T08:59:06.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T08:59:06.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T08:59:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:05 smithi114 ceph-mon[33315]: osdmap e60: 8 total, 8 up, 8 in 2024-09-16T08:59:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T08:59:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[29803]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[29803]: Saving service prometheus spec with placement smithi114=a;count:1 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[29803]: osdmap e60: 8 total, 8 up, 8 in 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[40056]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[40056]: Saving service prometheus spec with placement smithi114=a;count:1 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T08:59:06.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-16T08:59:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[40056]: osdmap e60: 8 total, 8 up, 8 in 2024-09-16T08:59:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T08:59:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-16T08:59:06.677 DEBUG:teuthology.orchestra.run.smithi111:node-exporter.a> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-16T08:59:06.679 DEBUG:teuthology.orchestra.run.smithi114:node-exporter.b> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-16T08:59:06.682 INFO:tasks.cephadm:Adding alertmanager.a on smithi111 2024-09-16T08:59:06.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi111=a' 2024-09-16T08:59:06.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: pgmap v198: 132 pgs: 5 creating+peering, 73 active+clean, 54 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-16T08:59:06.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:06.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T08:59:06.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T08:59:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: osdmap e61: 8 total, 8 up, 8 in 2024-09-16T08:59:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-16T08:59:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-16T08:59:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:06 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:59:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: pgmap v198: 132 pgs: 5 creating+peering, 73 active+clean, 54 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-16T08:59:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T08:59:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T08:59:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: osdmap e61: 8 total, 8 up, 8 in 2024-09-16T08:59:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: pgmap v198: 132 pgs: 5 creating+peering, 73 active+clean, 54 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/665948858' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2849567051' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: osdmap e61: 8 total, 8 up, 8 in 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:07.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:07.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-16T08:59:07.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-16T08:59:07.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:06 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:59:07.433 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:59:08.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:07 smithi114 ceph-mon[33315]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi111=a;smithi114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:08.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:07 smithi114 ceph-mon[33315]: Saving service node-exporter spec with placement smithi111=a;smithi114=b;count:2 2024-09-16T08:59:08.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:07 smithi114 ceph-mon[33315]: Deploying daemon iscsi.iscsi.a on smithi114 2024-09-16T08:59:08.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:07 smithi111 ceph-mon[29803]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi111=a;smithi114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:07 smithi111 ceph-mon[29803]: Saving service node-exporter spec with placement smithi111=a;smithi114=b;count:2 2024-09-16T08:59:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:07 smithi111 ceph-mon[29803]: Deploying daemon iscsi.iscsi.a on smithi114 2024-09-16T08:59:08.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:07 smithi111 ceph-mon[40056]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi111=a;smithi114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:08.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:07 smithi111 ceph-mon[40056]: Saving service node-exporter spec with placement smithi111=a;smithi114=b;count:2 2024-09-16T08:59:08.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:07 smithi111 ceph-mon[40056]: Deploying daemon iscsi.iscsi.a on smithi114 2024-09-16T08:59:08.526 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:08 smithi114 systemd[1]: Starting Ceph iscsi.iscsi.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T08:59:08.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.610+0000 7f355d235640 1 -- 172.21.15.114:0/1246219596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558103b60 msgr2=0x7f3558103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:08.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.610+0000 7f355d235640 1 --2- 172.21.15.114:0/1246219596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558103b60 0x7f3558103fe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3548009930 tx=0x7f354802a420 comp rx=0 tx=0).stop 2024-09-16T08:59:08.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.610+0000 7f355d235640 1 -- 172.21.15.114:0/1246219596 shutdown_connections 2024-09-16T08:59:08.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.610+0000 7f355d235640 1 --2- 172.21.15.114:0/1246219596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3558104520 0x7f355810aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.610+0000 7f355d235640 1 --2- 172.21.15.114:0/1246219596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558103b60 0x7f3558103fe0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.610+0000 7f355d235640 1 --2- 172.21.15.114:0/1246219596 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3558102a00 0x7f3558102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.610+0000 7f355d235640 1 -- 172.21.15.114:0/1246219596 >> 172.21.15.114:0/1246219596 conn(0x7f35580fe2f0 msgr2=0x7f3558100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:08.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.610+0000 7f355d235640 1 -- 172.21.15.114:0/1246219596 shutdown_connections 2024-09-16T08:59:08.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.610+0000 7f355d235640 1 -- 172.21.15.114:0/1246219596 wait complete. 2024-09-16T08:59:08.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355d235640 1 Processor -- start 2024-09-16T08:59:08.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355d235640 1 -- start start 2024-09-16T08:59:08.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355d235640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558102a00 0x7f35581984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:08.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355d235640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3558103b60 0x7f3558198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:08.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355d235640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3558104520 0x7f3558199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:08.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355d235640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f355810dc20 con 0x7f3558103b60 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355d235640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f355810daa0 con 0x7f3558102a00 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355d235640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f355810dda0 con 0x7f3558104520 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f3556d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558102a00 0x7f35581984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f3556d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558102a00 0x7f35581984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39168/0 (socket says 172.21.15.114:39168) 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f3556d9d640 1 -- 172.21.15.114:0/1252664861 learned_addr learned my addr 172.21.15.114:0/1252664861 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355659c640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3558103b60 0x7f3558198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.611+0000 7f355759e640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3558104520 0x7f3558199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f3556d9d640 1 -- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3558104520 msgr2=0x7f3558199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f3556d9d640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3558104520 0x7f3558199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f3556d9d640 1 -- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3558103b60 msgr2=0x7f3558198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f3556d9d640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3558103b60 0x7f3558198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f3556d9d640 1 -- 172.21.15.114:0/1252664861 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3558199fd0 con 0x7f3558102a00 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f355659c640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3558103b60 0x7f3558198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T08:59:08.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f3556d9d640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558102a00 0x7f35581984a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f354c00b4a0 tx=0x7f354c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f353ffff640 1 -- 172.21.15.114:0/1252664861 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c014070 con 0x7f3558102a00 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f353ffff640 1 -- 172.21.15.114:0/1252664861 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f354c010a60 con 0x7f3558102a00 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355819a2c0 con 0x7f3558102a00 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f353ffff640 1 -- 172.21.15.114:0/1252664861 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c0179e0 con 0x7f3558102a00 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.612+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35581a09a0 con 0x7f3558102a00 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.613+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3558102e80 con 0x7f3558102a00 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.613+0000 7f353ffff640 1 -- 172.21.15.114:0/1252664861 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f354c017070 con 0x7f3558102a00 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.614+0000 7f353ffff640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f354007ad20 0x7f354007d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.614+0000 7f353ffff640 1 -- 172.21.15.114:0/1252664861 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f354c09d590 con 0x7f3558102a00 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.614+0000 7f355659c640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f354007ad20 0x7f354007d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.615+0000 7f355659c640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f354007ad20 0x7f354007d1e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3548005ab0 tx=0x7f3548005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:08.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.618+0000 7f353ffff640 1 -- 172.21.15.114:0/1252664861 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f354c064570 con 0x7f3558102a00 2024-09-16T08:59:08.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.837+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}) -- 0x7f3558063590 con 0x7f354007ad20 2024-09-16T08:59:08.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.847+0000 7f353ffff640 1 -- 172.21.15.114:0/1252664861 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+33 (secure 0 0 0) 0x7f3558063590 con 0x7f354007ad20 2024-09-16T08:59:08.849 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled alertmanager update... 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f354007ad20 msgr2=0x7f354007d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f354007ad20 0x7f354007d1e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3548005ab0 tx=0x7f3548005fd0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558102a00 msgr2=0x7f35581984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558102a00 0x7f35581984a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f354c00b4a0 tx=0x7f354c00b960 comp rx=0 tx=0).stop 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 shutdown_connections 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f354007ad20 0x7f354007d1e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3558104520 0x7f3558199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3558103b60 0x7f3558198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 --2- 172.21.15.114:0/1252664861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3558102a00 0x7f35581984a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 >> 172.21.15.114:0/1252664861 conn(0x7f35580fe2f0 msgr2=0x7f35580fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:08.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 shutdown_connections 2024-09-16T08:59:08.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:08.849+0000 7f355d235640 1 -- 172.21.15.114:0/1252664861 wait complete. 2024-09-16T08:59:09.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:08 smithi114 ceph-mon[33315]: pgmap v201: 132 pgs: 3 creating+peering, 111 active+clean, 18 unknown; 580 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 4.7 KiB/s wr, 118 op/s 2024-09-16T08:59:09.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:08 smithi114 ceph-mon[33315]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T08:59:09.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:08 smithi114 ceph-mon[33315]: Cluster is now healthy 2024-09-16T08:59:09.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:08 smithi114 ceph-mon[33315]: osdmap e62: 8 total, 8 up, 8 in 2024-09-16T08:59:09.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:08 smithi114 podman[60444]: 2024-09-16 08:59:08.764206051 +0000 UTC m=+0.015662387 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T08:59:09.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:08 smithi114 podman[60444]: 2024-09-16 08:59:08.880701054 +0000 UTC m=+0.132157380 container create a0240c3768d6c630cfef2553f101a571da8e7a69ef596227f1e8b0aee3393150 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T08:59:09.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:08 smithi111 ceph-mon[29803]: pgmap v201: 132 pgs: 3 creating+peering, 111 active+clean, 18 unknown; 580 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 4.7 KiB/s wr, 118 op/s 2024-09-16T08:59:09.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:08 smithi111 ceph-mon[29803]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T08:59:09.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:08 smithi111 ceph-mon[29803]: Cluster is now healthy 2024-09-16T08:59:09.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:08 smithi111 ceph-mon[29803]: osdmap e62: 8 total, 8 up, 8 in 2024-09-16T08:59:09.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:08 smithi111 ceph-mon[40056]: pgmap v201: 132 pgs: 3 creating+peering, 111 active+clean, 18 unknown; 580 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 4.7 KiB/s wr, 118 op/s 2024-09-16T08:59:09.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:08 smithi111 ceph-mon[40056]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T08:59:09.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:08 smithi111 ceph-mon[40056]: Cluster is now healthy 2024-09-16T08:59:09.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:08 smithi111 ceph-mon[40056]: osdmap e62: 8 total, 8 up, 8 in 2024-09-16T08:59:09.576 DEBUG:teuthology.orchestra.run.smithi111:alertmanager.a> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-16T08:59:09.579 INFO:tasks.cephadm:Adding grafana.a on smithi114 2024-09-16T08:59:09.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi114=a' 2024-09-16T08:59:09.801 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:09 smithi114 podman[60444]: 2024-09-16 08:59:09.707432989 +0000 UTC m=+0.958889327 container init a0240c3768d6c630cfef2553f101a571da8e7a69ef596227f1e8b0aee3393150 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-16T08:59:09.801 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:09 smithi114 podman[60444]: 2024-09-16 08:59:09.712169579 +0000 UTC m=+0.963625913 container start a0240c3768d6c630cfef2553f101a571da8e7a69ef596227f1e8b0aee3393150 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-16T08:59:09.801 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:09 smithi114 bash[60444]: a0240c3768d6c630cfef2553f101a571da8e7a69ef596227f1e8b0aee3393150 2024-09-16T08:59:09.801 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:09 smithi114 systemd[1]: Started Ceph iscsi.iscsi.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:59:09.865 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:59:10.138 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:09 smithi114 ceph-mon[33315]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:10.139 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:09 smithi114 ceph-mon[33315]: Saving service alertmanager spec with placement smithi111=a;count:1 2024-09-16T08:59:10.139 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:09 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:10.139 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug Started the configuration object watcher 2024-09-16T08:59:10.139 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug Checking for config object changes every 1s 2024-09-16T08:59:10.139 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug Processing osd blocklist entries for this node 2024-09-16T08:59:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:09 smithi111 ceph-mon[29803]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:09 smithi111 ceph-mon[29803]: Saving service alertmanager spec with placement smithi111=a;count:1 2024-09-16T08:59:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:09 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:10.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:09 smithi111 ceph-mon[40056]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:10.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:09 smithi111 ceph-mon[40056]: Saving service alertmanager spec with placement smithi111=a;count:1 2024-09-16T08:59:10.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:09 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:10.497 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug Reading the configuration object to update local LIO configuration 2024-09-16T08:59:10.497 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug Configuration does not have an entry for this host(smithi114.front.sepia.ceph.com) - nothing to define to LIO 2024-09-16T08:59:10.497 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-16T08:59:10.497 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: * Environment: production 2024-09-16T08:59:10.497 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-16T08:59:10.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: Use a production WSGI server instead. 2024-09-16T08:59:10.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: * Debug mode: off 2024-09-16T08:59:10.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug * Running on all addresses. 2024-09-16T08:59:10.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-16T08:59:10.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: * Running on all addresses. 2024-09-16T08:59:10.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-16T08:59:10.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug * Running on http://[fe80::ec4:7aff:fe8f:cb63]:5000/ (Press CTRL+C to quit) 2024-09-16T08:59:10.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: * Running on http://[fe80::ec4:7aff:fe8f:cb63]:5000/ (Press CTRL+C to quit) 2024-09-16T08:59:10.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.949+0000 7f7b94d86640 1 -- 172.21.15.114:0/1212016114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90102a70 msgr2=0x7f7b90102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.949+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1212016114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90102a70 0x7f7b90102ef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009930 tx=0x7f7b8402a420 comp rx=0 tx=0).stop 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.949+0000 7f7b94d86640 1 -- 172.21.15.114:0/1212016114 shutdown_connections 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.949+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1212016114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b90103430 0x7f7b90109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.949+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1212016114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90102a70 0x7f7b90102ef0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.949+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1212016114 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b90101910 0x7f7b90101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.949+0000 7f7b94d86640 1 -- 172.21.15.114:0/1212016114 >> 172.21.15.114:0/1212016114 conn(0x7f7b900fd200 msgr2=0x7f7b900ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.949+0000 7f7b94d86640 1 -- 172.21.15.114:0/1212016114 shutdown_connections 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.950+0000 7f7b94d86640 1 -- 172.21.15.114:0/1212016114 wait complete. 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.950+0000 7f7b94d86640 1 Processor -- start 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.950+0000 7f7b94d86640 1 -- start start 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.950+0000 7f7b94d86640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b90101910 0x7f7b90198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.950+0000 7f7b94d86640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b90102a70 0x7f7b90198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.950+0000 7f7b94d86640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90103430 0x7f7b901997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.950+0000 7f7b94d86640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9010cae0 con 0x7f7b90101910 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.950+0000 7f7b94d86640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9010c960 con 0x7f7b90103430 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.950+0000 7f7b94d86640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f7b9010cc60 con 0x7f7b90102a70 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90103430 0x7f7b901997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90103430 0x7f7b901997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39240/0 (socket says 172.21.15.114:39240) 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8ed9d640 1 -- 172.21.15.114:0/1920651707 learned_addr learned my addr 172.21.15.114:0/1920651707 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8dd9b640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b90102a70 0x7f7b90198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8e59c640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b90101910 0x7f7b90198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8ed9d640 1 -- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b90102a70 msgr2=0x7f7b90198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8ed9d640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b90102a70 0x7f7b90198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8ed9d640 1 -- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b90101910 msgr2=0x7f7b90198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8ed9d640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b90101910 0x7f7b90198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8ed9d640 1 -- 172.21.15.114:0/1920651707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9019a080 con 0x7f7b90103430 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8e59c640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b90101910 0x7f7b90198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b8ed9d640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90103430 0x7f7b901997e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800b970 tx=0x7f7b8800be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b7f7fe640 1 -- 172.21.15.114:0/1920651707 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88019070 con 0x7f7b90103430 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b9019a370 con 0x7f7b90103430 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b7f7fe640 1 -- 172.21.15.114:0/1920651707 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f7b88014440 con 0x7f7b90103430 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b7f7fe640 1 -- 172.21.15.114:0/1920651707 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88014e60 con 0x7f7b90103430 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.951+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b901a0a50 con 0x7f7b90103430 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.952+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b90101d90 con 0x7f7b90103430 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.954+0000 7f7b7f7fe640 1 -- 172.21.15.114:0/1920651707 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 103080+0+0 (secure 0 0 0) 0x7f7b8801c070 con 0x7f7b90103430 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.954+0000 7f7b7f7fe640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7b7807ad20 0x7f7b7807d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.954+0000 7f7b7f7fe640 1 -- 172.21.15.114:0/1920651707 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f7b8809d6d0 con 0x7f7b90103430 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.954+0000 7f7b8e59c640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7b7807ad20 0x7f7b7807d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:10.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.955+0000 7f7b8e59c640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7b7807ad20 0x7f7b7807d1e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009930 tx=0x7f7b80008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:10.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:10.962+0000 7f7b7f7fe640 1 -- 172.21.15.114:0/1920651707 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f7b88068a90 con 0x7f7b90103430 2024-09-16T08:59:11.139 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-mon[33315]: pgmap v203: 132 pgs: 126 active+clean, 6 unknown; 581 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 6.7 KiB/s wr, 182 op/s 2024-09-16T08:59:11.139 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.139 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.139 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.140 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-mon[33315]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-16T08:59:11.140 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.140 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/1972493653' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-16T08:59:11.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.166+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 --> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}) -- 0x7f7b9002e910 con 0x7f7b7807ad20 2024-09-16T08:59:11.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.173+0000 7f7b7f7fe640 1 -- 172.21.15.114:0/1920651707 <== mgr.14152 v2:172.21.15.111:6800/3306957597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f7b9002e910 con 0x7f7b7807ad20 2024-09-16T08:59:11.175 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled grafana update... 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[29803]: pgmap v203: 132 pgs: 126 active+clean, 6 unknown; 581 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 6.7 KiB/s wr, 182 op/s 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[29803]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/1972493653' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[40056]: pgmap v203: 132 pgs: 126 active+clean, 6 unknown; 581 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 6.7 KiB/s wr, 182 op/s 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[40056]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-16T08:59:11.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:11.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/1972493653' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7b7807ad20 msgr2=0x7f7b7807d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7b7807ad20 0x7f7b7807d1e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009930 tx=0x7f7b80008040 comp rx=0 tx=0).stop 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90103430 msgr2=0x7f7b901997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90103430 0x7f7b901997e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800b970 tx=0x7f7b8800be30 comp rx=0 tx=0).stop 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 shutdown_connections 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7b7807ad20 0x7f7b7807d1e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b90103430 0x7f7b901997e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b90102a70 0x7f7b90198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 --2- 172.21.15.114:0/1920651707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b90101910 0x7f7b90198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 >> 172.21.15.114:0/1920651707 conn(0x7f7b900fd200 msgr2=0x7f7b9006bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 shutdown_connections 2024-09-16T08:59:11.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:11.176+0000 7f7b94d86640 1 -- 172.21.15.114:0/1920651707 wait complete. 2024-09-16T08:59:11.838 DEBUG:teuthology.orchestra.run.smithi114:grafana.a> sudo journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@grafana.a.service 2024-09-16T08:59:11.841 INFO:tasks.cephadm:Setting up client nodes... 2024-09-16T08:59:11.841 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-16T08:59:12.096 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:12.133 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:11 smithi114 ceph-mon[33315]: Deploying daemon prometheus.a on smithi114 2024-09-16T08:59:12.133 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:11 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:12.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:11 smithi111 ceph-mon[29803]: Deploying daemon prometheus.a on smithi114 2024-09-16T08:59:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:11 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:12.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:11 smithi111 ceph-mon[40056]: Deploying daemon prometheus.a on smithi114 2024-09-16T08:59:12.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:11 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.585+0000 7fca30194640 1 -- 172.21.15.111:0/2739959697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca28105bf0 msgr2=0x7fca28107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.585+0000 7fca30194640 1 --2- 172.21.15.111:0/2739959697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca28105bf0 0x7fca28107fe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fca18009900 tx=0x7fca1802a410 comp rx=0 tx=0).stop 2024-09-16T08:59:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.585+0000 7fca30194640 1 -- 172.21.15.111:0/2739959697 shutdown_connections 2024-09-16T08:59:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.585+0000 7fca30194640 1 --2- 172.21.15.111:0/2739959697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca28108720 0x7fca2810abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.585+0000 7fca30194640 1 --2- 172.21.15.111:0/2739959697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca28105bf0 0x7fca28107fe0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.585+0000 7fca30194640 1 --2- 172.21.15.111:0/2739959697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca2806bfd0 0x7fca281056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.585+0000 7fca30194640 1 -- 172.21.15.111:0/2739959697 >> 172.21.15.111:0/2739959697 conn(0x7fca280fd200 msgr2=0x7fca280ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.585+0000 7fca30194640 1 -- 172.21.15.111:0/2739959697 shutdown_connections 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.585+0000 7fca30194640 1 -- 172.21.15.111:0/2739959697 wait complete. 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca30194640 1 Processor -- start 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca30194640 1 -- start start 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca30194640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca2806bfd0 0x7fca28198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca30194640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca28105bf0 0x7fca281989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca30194640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca28108720 0x7fca281995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca30194640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fca2810db30 con 0x7fca28108720 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca30194640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca2810d9b0 con 0x7fca28105bf0 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca30194640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fca2810dcb0 con 0x7fca2806bfd0 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca2df30640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca2806bfd0 0x7fca28198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca2df30640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca2806bfd0 0x7fca28198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:47752/0 (socket says 172.21.15.111:47752) 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca2df30640 1 -- 172.21.15.111:0/2832554102 learned_addr learned my addr 172.21.15.111:0/2832554102 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca2e731640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca28108720 0x7fca281995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca2d72f640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca28105bf0 0x7fca281989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.586+0000 7fca2df30640 1 -- 172.21.15.111:0/2832554102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca28105bf0 msgr2=0x7fca281989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca2df30640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca28105bf0 0x7fca281989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca2df30640 1 -- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca28108720 msgr2=0x7fca281995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca2df30640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca28108720 0x7fca281995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca2df30640 1 -- 172.21.15.111:0/2832554102 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca28199e80 con 0x7fca2806bfd0 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca2df30640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca2806bfd0 0x7fca28198350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fca2400ea60 tx=0x7fca2400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca1effd640 1 -- 172.21.15.111:0/2832554102 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2400cde0 con 0x7fca2806bfd0 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca1effd640 1 -- 172.21.15.111:0/2832554102 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fca24004470 con 0x7fca2806bfd0 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2819a170 con 0x7fca2806bfd0 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca1effd640 1 -- 172.21.15.111:0/2832554102 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24010430 con 0x7fca2806bfd0 2024-09-16T08:59:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.587+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fca281a0850 con 0x7fca2806bfd0 2024-09-16T08:59:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.588+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca28105fd0 con 0x7fca2806bfd0 2024-09-16T08:59:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.589+0000 7fca1effd640 1 -- 172.21.15.111:0/2832554102 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 16) ==== 103131+0+0 (secure 0 0 0) 0x7fca24003f60 con 0x7fca2806bfd0 2024-09-16T08:59:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.589+0000 7fca1effd640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fca1407adb0 0x7fca1407d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.589+0000 7fca2d72f640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fca1407adb0 0x7fca1407d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.589+0000 7fca1effd640 1 -- 172.21.15.111:0/2832554102 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7fca2409e500 con 0x7fca2806bfd0 2024-09-16T08:59:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.589+0000 7fca2d72f640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fca1407adb0 0x7fca1407d270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fca18005ab0 tx=0x7fca18005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:12.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.594+0000 7fca1effd640 1 -- 172.21.15.111:0/2832554102 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fca24069840 con 0x7fca2806bfd0 2024-09-16T08:59:12.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.837+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fca2802e8f0 con 0x7fca2806bfd0 2024-09-16T08:59:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.842+0000 7fca1effd640 1 -- 172.21.15.111:0/2832554102 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) ==== 170+0+59 (secure 0 0 0) 0x7fca2406e6f0 con 0x7fca2806bfd0 2024-09-16T08:59:12.843 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2024-09-16T08:59:12.843 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBg8+dmAGQAMhAA/rffwXThYa/SsOdu3iaifA== 2024-09-16T08:59:12.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fca1407adb0 msgr2=0x7fca1407d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:12.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fca1407adb0 0x7fca1407d270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fca18005ab0 tx=0x7fca18005a20 comp rx=0 tx=0).stop 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca2806bfd0 msgr2=0x7fca28198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca2806bfd0 0x7fca28198350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fca2400ea60 tx=0x7fca2400ef20 comp rx=0 tx=0).stop 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 shutdown_connections 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fca1407adb0 0x7fca1407d270 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca28108720 0x7fca281995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca28105bf0 0x7fca281989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 --2- 172.21.15.111:0/2832554102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca2806bfd0 0x7fca28198350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 >> 172.21.15.111:0/2832554102 conn(0x7fca280fd200 msgr2=0x7fca28103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 shutdown_connections 2024-09-16T08:59:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:12.844+0000 7fca30194640 1 -- 172.21.15.111:0/2832554102 wait complete. 2024-09-16T08:59:13.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:12 smithi114 ceph-mon[33315]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 6.0 KiB/s wr, 171 op/s 2024-09-16T08:59:13.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:12 smithi114 ceph-mon[33315]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:13.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:12 smithi114 ceph-mon[33315]: Saving service grafana spec with placement smithi114=a;count:1 2024-09-16T08:59:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:12 smithi114 ceph-mon[33315]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-16T08:59:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:12 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2832554102' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T08:59:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[29803]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 6.0 KiB/s wr, 171 op/s 2024-09-16T08:59:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[29803]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[29803]: Saving service grafana spec with placement smithi114=a;count:1 2024-09-16T08:59:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[29803]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-16T08:59:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2832554102' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T08:59:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[40056]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 6.0 KiB/s wr, 171 op/s 2024-09-16T08:59:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[40056]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T08:59:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[40056]: Saving service grafana spec with placement smithi114=a;count:1 2024-09-16T08:59:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[40056]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-16T08:59:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2832554102' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T08:59:13.651 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T08:59:13.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-16T08:59:13.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T08:59:13.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-16T08:59:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T08:59:14.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.633+0000 7f6abb2df640 1 -- 172.21.15.114:0/2154862666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4103b60 msgr2=0x7f6ab4103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.633+0000 7f6abb2df640 1 --2- 172.21.15.114:0/2154862666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4103b60 0x7f6ab4103fe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009930 tx=0x7f6aa402a420 comp rx=0 tx=0).stop 2024-09-16T08:59:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.633+0000 7f6abb2df640 1 -- 172.21.15.114:0/2154862666 shutdown_connections 2024-09-16T08:59:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.633+0000 7f6abb2df640 1 --2- 172.21.15.114:0/2154862666 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ab4104520 0x7f6ab410aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.633+0000 7f6abb2df640 1 --2- 172.21.15.114:0/2154862666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4103b60 0x7f6ab4103fe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.633+0000 7f6abb2df640 1 --2- 172.21.15.114:0/2154862666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab4102a00 0x7f6ab4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.633+0000 7f6abb2df640 1 -- 172.21.15.114:0/2154862666 >> 172.21.15.114:0/2154862666 conn(0x7f6ab40fe2f0 msgr2=0x7f6ab4100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.633+0000 7f6abb2df640 1 -- 172.21.15.114:0/2154862666 shutdown_connections 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.633+0000 7f6abb2df640 1 -- 172.21.15.114:0/2154862666 wait complete. 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6abb2df640 1 Processor -- start 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6abb2df640 1 -- start start 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6abb2df640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab4102a00 0x7f6ab4198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6abb2df640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ab4103b60 0x7f6ab41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6abb2df640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4104520 0x7f6ab41995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6abb2df640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab410daf0 con 0x7f6ab4102a00 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6abb2df640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab410d970 con 0x7f6ab4104520 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6abb2df640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab410dc70 con 0x7f6ab4103b60 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab987c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4104520 0x7f6ab41995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab987c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4104520 0x7f6ab41995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36498/0 (socket says 172.21.15.114:36498) 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab987c640 1 -- 172.21.15.114:0/1540432387 learned_addr learned my addr 172.21.15.114:0/1540432387 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab887a640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ab4103b60 0x7f6ab41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab907b640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab4102a00 0x7f6ab4198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab987c640 1 -- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ab4103b60 msgr2=0x7f6ab41989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab987c640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ab4103b60 0x7f6ab41989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab987c640 1 -- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab4102a00 msgr2=0x7f6ab4198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab987c640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab4102a00 0x7f6ab4198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab987c640 1 -- 172.21.15.114:0/1540432387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab4199e80 con 0x7f6ab4104520 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.634+0000 7f6ab907b640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab4102a00 0x7f6ab4198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.635+0000 7f6ab987c640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4104520 0x7f6ab41995e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00cad0 tx=0x7f6aac00cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.635+0000 7f6aaa7fc640 1 -- 172.21.15.114:0/1540432387 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac00ae20 con 0x7f6ab4104520 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.635+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab419a170 con 0x7f6ab4104520 2024-09-16T08:59:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.635+0000 7f6aaa7fc640 1 -- 172.21.15.114:0/1540432387 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f6aac005020 con 0x7f6ab4104520 2024-09-16T08:59:14.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.635+0000 7f6aaa7fc640 1 -- 172.21.15.114:0/1540432387 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac00f6b0 con 0x7f6ab4104520 2024-09-16T08:59:14.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.635+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab41a0850 con 0x7f6ab4104520 2024-09-16T08:59:14.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.636+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ab4102e00 con 0x7f6ab4104520 2024-09-16T08:59:14.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.636+0000 7f6aaa7fc640 1 -- 172.21.15.114:0/1540432387 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 103131+0+0 (secure 0 0 0) 0x7f6aac004a40 con 0x7f6ab4104520 2024-09-16T08:59:14.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.637+0000 7f6aaa7fc640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f6aa007adb0 0x7f6aa007d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:14.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.637+0000 7f6aaa7fc640 1 -- 172.21.15.114:0/1540432387 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f6aac09e390 con 0x7f6ab4104520 2024-09-16T08:59:14.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.637+0000 7f6ab907b640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f6aa007adb0 0x7f6aa007d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:14.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.638+0000 7f6ab907b640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f6aa007adb0 0x7f6aa007d270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00058c0 tx=0x7f6ab0005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:14.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.641+0000 7f6aaa7fc640 1 -- 172.21.15.114:0/1540432387 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f6aac014030 con 0x7f6ab4104520 2024-09-16T08:59:14.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.885+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f6ab402e7d0 con 0x7f6ab4104520 2024-09-16T08:59:14.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.892+0000 7f6aaa7fc640 1 -- 172.21.15.114:0/1540432387 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (secure 0 0 0) 0x7f6aac065270 con 0x7f6ab4104520 2024-09-16T08:59:14.892 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:14 smithi114 ceph-mon[33315]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.7 KiB/s wr, 146 op/s 2024-09-16T08:59:14.893 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2024-09-16T08:59:14.893 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBi8+dm2xvlNBAAjt85gkjCXTZ9DXnwnnV3dw== 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f6aa007adb0 msgr2=0x7f6aa007d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f6aa007adb0 0x7f6aa007d270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00058c0 tx=0x7f6ab0005ec0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4104520 msgr2=0x7f6ab41995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4104520 0x7f6ab41995e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00cad0 tx=0x7f6aac00cf90 comp rx=0 tx=0).stop 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 shutdown_connections 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f6aa007adb0 0x7f6aa007d270 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab4104520 0x7f6ab41995e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ab4103b60 0x7f6ab41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 --2- 172.21.15.114:0/1540432387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab4102a00 0x7f6ab4198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 >> 172.21.15.114:0/1540432387 conn(0x7f6ab40fe2f0 msgr2=0x7f6ab40fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:14.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 shutdown_connections 2024-09-16T08:59:14.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-16T08:59:14.894+0000 7f6abb2df640 1 -- 172.21.15.114:0/1540432387 wait complete. 2024-09-16T08:59:15.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:14 smithi111 ceph-mon[29803]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.7 KiB/s wr, 146 op/s 2024-09-16T08:59:15.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:14 smithi111 ceph-mon[40056]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.7 KiB/s wr, 146 op/s 2024-09-16T08:59:15.936 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-16T08:59:15.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-16T08:59:15.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-16T08:59:15.978 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-16T08:59:16.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.114:0/1540432387' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:16.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:16.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T08:59:16.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.114:0/1540432387' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:16.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T08:59:16.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.114:0/1540432387' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:16.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-16T08:59:16.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T08:59:16.237 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:16.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 -- 172.21.15.111:0/2803738306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34a8108790 msgr2=0x7f34a8108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 --2- 172.21.15.111:0/2803738306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34a8108790 0x7f34a8108bd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3490009860 tx=0x7f349002a340 comp rx=0 tx=0).stop 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 -- 172.21.15.111:0/2803738306 shutdown_connections 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 --2- 172.21.15.111:0/2803738306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34a8109110 0x7f34a810f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 --2- 172.21.15.111:0/2803738306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34a8108790 0x7f34a8108bd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 --2- 172.21.15.111:0/2803738306 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34a8102d80 0x7f34a8103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 -- 172.21.15.111:0/2803738306 >> 172.21.15.111:0/2803738306 conn(0x7f34a80fe7d0 msgr2=0x7f34a8100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 -- 172.21.15.111:0/2803738306 shutdown_connections 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 -- 172.21.15.111:0/2803738306 wait complete. 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.717+0000 7f34accba640 1 Processor -- start 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34accba640 1 -- start start 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34accba640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34a8102d80 0x7f34a80754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34accba640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34a8108790 0x7f34a80759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:16.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34accba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34a8109110 0x7f34a8075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34accba640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f34a8112230 con 0x7f34a8108790 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34accba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f34a81120b0 con 0x7f34a8109110 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34accba640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f34a81123b0 con 0x7f34a8102d80 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a659c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34a8102d80 0x7f34a80754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a659c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34a8102d80 0x7f34a80754a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:47768/0 (socket says 172.21.15.111:47768) 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a5d9b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34a8108790 0x7f34a80759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a659c640 1 -- 172.21.15.111:0/2115109252 learned_addr learned my addr 172.21.15.111:0/2115109252 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a6d9d640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34a8109110 0x7f34a8075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a659c640 1 -- 172.21.15.111:0/2115109252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34a8109110 msgr2=0x7f34a8075f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a659c640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34a8109110 0x7f34a8075f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a659c640 1 -- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34a8108790 msgr2=0x7f34a80759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a659c640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34a8108790 0x7f34a80759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a659c640 1 -- 172.21.15.111:0/2115109252 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a81a6bc0 con 0x7f34a8102d80 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a6d9d640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34a8109110 0x7f34a8075f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f34a659c640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34a8102d80 0x7f34a80754a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f349800e9c0 tx=0x7f349800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f349f7fe640 1 -- 172.21.15.111:0/2115109252 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349800cde0 con 0x7f34a8102d80 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.718+0000 7f349f7fe640 1 -- 172.21.15.111:0/2115109252 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f3498004470 con 0x7f34a8102d80 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.719+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a81a6eb0 con 0x7f34a8102d80 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.719+0000 7f349f7fe640 1 -- 172.21.15.111:0/2115109252 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3498003e50 con 0x7f34a8102d80 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.719+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a81a7360 con 0x7f34a8102d80 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.720+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a8103160 con 0x7f34a8102d80 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.720+0000 7f349f7fe640 1 -- 172.21.15.111:0/2115109252 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 16) ==== 103131+0+0 (secure 0 0 0) 0x7f3498004000 con 0x7f34a8102d80 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.721+0000 7f349f7fe640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f349407af50 0x7f349407d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.721+0000 7f349f7fe640 1 -- 172.21.15.111:0/2115109252 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f349809e6e0 con 0x7f34a8102d80 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.721+0000 7f34a5d9b640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f349407af50 0x7f349407d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:16.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.721+0000 7f34a5d9b640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f349407af50 0x7f349407d410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f349002a930 tx=0x7f34900058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:16.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.725+0000 7f349f7fe640 1 -- 172.21.15.111:0/2115109252 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f34980655c0 con 0x7f34a8102d80 2024-09-16T08:59:16.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.901+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f34a810aef0 con 0x7f34a8102d80 2024-09-16T08:59:16.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:16 smithi111 ceph-mon[29803]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 130 op/s 2024-09-16T08:59:16.904 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:16 smithi111 ceph-mon[40056]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 130 op/s 2024-09-16T08:59:16.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.902+0000 7f349f7fe640 1 -- 172.21.15.111:0/2115109252 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v23) ==== 86+0+187 (secure 0 0 0) 0x7f34980699a0 con 0x7f34a8102d80 2024-09-16T08:59:16.905 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:59:16.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f349407af50 msgr2=0x7f349407d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:16.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f349407af50 0x7f349407d410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f349002a930 tx=0x7f34900058d0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34a8102d80 msgr2=0x7f34a80754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:16.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34a8102d80 0x7f34a80754a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f349800e9c0 tx=0x7f349800ee80 comp rx=0 tx=0).stop 2024-09-16T08:59:16.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 shutdown_connections 2024-09-16T08:59:16.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f349407af50 0x7f349407d410 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34a8109110 0x7f34a8075f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34a8108790 0x7f34a80759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 --2- 172.21.15.111:0/2115109252 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34a8102d80 0x7f34a80754a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:16.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 >> 172.21.15.111:0/2115109252 conn(0x7f34a80fe7d0 msgr2=0x7f34a8100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:16.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 shutdown_connections 2024-09-16T08:59:16.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:16.904+0000 7f34accba640 1 -- 172.21.15.111:0/2115109252 wait complete. 2024-09-16T08:59:17.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:16 smithi114 ceph-mon[33315]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 130 op/s 2024-09-16T08:59:17.586 INFO:teuthology.orchestra.run.smithi111.stdout:[{"version":23,"timestamp":"2024-09-16T08:59:06.491206+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/RGW_API_SECRET_KEY","new_value":"vQe6qBvs6k8JR4f9aCugMQ1ksE29vOMhdSiW8X7I"}]}] 2024-09-16T08:59:17.586 INFO:tasks.ceph_manager:config epoch is 23 2024-09-16T08:59:17.586 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T08:59:17.586 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-16T08:59:17.586 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-16T08:59:17.833 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:18.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2115109252' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-16T08:59:18.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:17 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:18.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2115109252' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-16T08:59:18.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:17 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:18.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2115109252' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-16T08:59:18.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:17 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:18.293 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 -- 172.21.15.111:0/1731046345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 msgr2=0x7fb0e8102060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 --2- 172.21.15.111:0/1731046345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 0x7fb0e8102060 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009860 tx=0x7fb0d802a340 comp rx=0 tx=0).stop 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 -- 172.21.15.111:0/1731046345 shutdown_connections 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 --2- 172.21.15.111:0/1731046345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8108010 0x7fb0e810e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 --2- 172.21.15.111:0/1731046345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e8107690 0x7fb0e8107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 --2- 172.21.15.111:0/1731046345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 0x7fb0e8102060 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 -- 172.21.15.111:0/1731046345 >> 172.21.15.111:0/1731046345 conn(0x7fb0e80fd6d0 msgr2=0x7fb0e80ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 -- 172.21.15.111:0/1731046345 shutdown_connections 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 -- 172.21.15.111:0/1731046345 wait complete. 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.288+0000 7fb0ee408640 1 Processor -- start 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0ee408640 1 -- start start 2024-09-16T08:59:18.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0ee408640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 0x7fb0e81987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0ee408640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e8107690 0x7fb0e8198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0ee408640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8108010 0x7fb0e819e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0ee408640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81122a0 con 0x7fb0e8107690 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0ee408640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e8112120 con 0x7fb0e8108010 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0ee408640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e8112420 con 0x7fb0e8101c80 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e7fff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 0x7fb0e81987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e7fff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 0x7fb0e81987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:47788/0 (socket says 172.21.15.111:47788) 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e7fff640 1 -- 172.21.15.111:0/3633985628 learned_addr learned my addr 172.21.15.111:0/3633985628 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e77fe640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e8107690 0x7fb0e8198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0ec9a5640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8108010 0x7fb0e819e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e7fff640 1 -- 172.21.15.111:0/3633985628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8108010 msgr2=0x7fb0e819e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e7fff640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8108010 0x7fb0e819e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e7fff640 1 -- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e8107690 msgr2=0x7fb0e8198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:18.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e7fff640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e8107690 0x7fb0e8198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e7fff640 1 -- 172.21.15.111:0/3633985628 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e819e880 con 0x7fb0e8101c80 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.289+0000 7fb0e7fff640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 0x7fb0e81987d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80057c0 tx=0x7fb0d8033d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.290+0000 7fb0e57fa640 1 -- 172.21.15.111:0/3633985628 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8037040 con 0x7fb0e8101c80 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.290+0000 7fb0e57fa640 1 -- 172.21.15.111:0/3633985628 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fb0d8040070 con 0x7fb0e8101c80 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.290+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e819eb10 con 0x7fb0e8101c80 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.290+0000 7fb0e57fa640 1 -- 172.21.15.111:0/3633985628 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8004490 con 0x7fb0e8101c80 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.290+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e81a5230 con 0x7fb0e8101c80 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.291+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e8102060 con 0x7fb0e8101c80 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.291+0000 7fb0e57fa640 1 -- 172.21.15.111:0/3633985628 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 16) ==== 103131+0+0 (secure 0 0 0) 0x7fb0d8030920 con 0x7fb0e8101c80 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.292+0000 7fb0e57fa640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d407adb0 0x7fb0d407d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.292+0000 7fb0e57fa640 1 -- 172.21.15.111:0/3633985628 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7fb0d80bdf90 con 0x7fb0e8101c80 2024-09-16T08:59:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.292+0000 7fb0e77fe640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d407adb0 0x7fb0d407d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:18.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.292+0000 7fb0e77fe640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d407adb0 0x7fb0d407d270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8107240 tx=0x7fb0d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.296+0000 7fb0e57fa640 1 -- 172.21.15.111:0/3633985628 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fb0d8084ef0 con 0x7fb0e8101c80 2024-09-16T08:59:18.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.519+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb0e8109df0 con 0x7fb0e8101c80 2024-09-16T08:59:18.523 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.522+0000 7fb0e57fa640 1 -- 172.21.15.111:0/3633985628 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) ==== 74+0+197795 (secure 0 0 0) 0x7fb0d80892d0 con 0x7fb0e8101c80 2024-09-16T08:59:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:59:18.528 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d407adb0 msgr2=0x7fb0d407d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:18.528 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d407adb0 0x7fb0d407d270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8107240 tx=0x7fb0d0009260 comp rx=0 tx=0).stop 2024-09-16T08:59:18.528 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 msgr2=0x7fb0e81987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:18.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 0x7fb0e81987d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80057c0 tx=0x7fb0d8033d00 comp rx=0 tx=0).stop 2024-09-16T08:59:18.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 shutdown_connections 2024-09-16T08:59:18.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fb0d407adb0 0x7fb0d407d270 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:18.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8108010 0x7fb0e819e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:18.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0e8107690 0x7fb0e8198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:18.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 --2- 172.21.15.111:0/3633985628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0e8101c80 0x7fb0e81987d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:18.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 >> 172.21.15.111:0/3633985628 conn(0x7fb0e80fd6d0 msgr2=0x7fb0e80ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:18.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 shutdown_connections 2024-09-16T08:59:18.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:18.526+0000 7fb0ee408640 1 -- 172.21.15.111:0/3633985628 wait complete. 2024-09-16T08:59:19.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:18 smithi114 ceph-mon[33315]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-09-16T08:59:19.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3633985628' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-16T08:59:19.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:18 smithi111 ceph-mon[29803]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-09-16T08:59:19.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3633985628' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-16T08:59:19.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:18 smithi111 ceph-mon[40056]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-09-16T08:59:19.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3633985628' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-16T08:59:19.208 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3306957597},{"type":"v1","addr":"172.21.15.111:6801","nonce":3306957597}]},"active_addr":"172.21.15.111:6801/3306957597","active_change":"2024-09-16T08:54:06.869957+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.111:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":195187973}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":2013099899}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":744880041}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":3220801815}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":1635160438}]}]} 2024-09-16T08:59:19.213 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-16T08:59:19.213 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-16T08:59:19.214 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T08:59:19.460 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.913+0000 7f4b8cf92640 1 -- 172.21.15.111:0/2114223715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 msgr2=0x7f4b88108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.913+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/2114223715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 0x7f4b88108bd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009900 tx=0x7f4b7c02a3f0 comp rx=0 tx=0).stop 2024-09-16T08:59:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.913+0000 7f4b8cf92640 1 -- 172.21.15.111:0/2114223715 shutdown_connections 2024-09-16T08:59:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.913+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/2114223715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b88109110 0x7f4b8810f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.913+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/2114223715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 0x7f4b88108bd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.913+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/2114223715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b88102d80 0x7f4b88103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.913+0000 7f4b8cf92640 1 -- 172.21.15.111:0/2114223715 >> 172.21.15.111:0/2114223715 conn(0x7f4b880fe7d0 msgr2=0x7f4b88100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.913+0000 7f4b8cf92640 1 -- 172.21.15.111:0/2114223715 shutdown_connections 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.913+0000 7f4b8cf92640 1 -- 172.21.15.111:0/2114223715 wait complete. 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8cf92640 1 Processor -- start 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8cf92640 1 -- start start 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8cf92640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b88102d80 0x7f4b881987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8cf92640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 0x7f4b88198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8cf92640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b88109110 0x7f4b8819e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8cf92640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b88112280 con 0x7f4b88102d80 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8cf92640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b88112100 con 0x7f4b88109110 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8cf92640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f4b88112400 con 0x7f4b88108790 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b88102d80 0x7f4b881987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b85d9b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 0x7f4b88198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b85d9b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 0x7f4b88198d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:48366/0 (socket says 172.21.15.111:48366) 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b85d9b640 1 -- 172.21.15.111:0/210752195 learned_addr learned my addr 172.21.15.111:0/210752195 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.914+0000 7f4b8659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b88102d80 0x7f4b881987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45102/0 (socket says 172.21.15.111:45102) 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b86d9d640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b88109110 0x7f4b8819e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b8659c640 1 -- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 msgr2=0x7f4b88198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b8659c640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 0x7f4b88198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b8659c640 1 -- 172.21.15.111:0/210752195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b88109110 msgr2=0x7f4b8819e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b8659c640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b88109110 0x7f4b8819e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b8659c640 1 -- 172.21.15.111:0/210752195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8819e870 con 0x7f4b88102d80 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b85d9b640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 0x7f4b88198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b8659c640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b88102d80 0x7f4b881987c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800e9f0 tx=0x7f4b7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b777fe640 1 -- 172.21.15.111:0/210752195 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7800cd80 con 0x7f4b88102d80 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8819eb60 con 0x7f4b88102d80 2024-09-16T08:59:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b777fe640 1 -- 172.21.15.111:0/210752195 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f4b78004470 con 0x7f4b88102d80 2024-09-16T08:59:19.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b777fe640 1 -- 172.21.15.111:0/210752195 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78003e10 con 0x7f4b88102d80 2024-09-16T08:59:19.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.915+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b88075550 con 0x7f4b88102d80 2024-09-16T08:59:19.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.916+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b88104030 con 0x7f4b88102d80 2024-09-16T08:59:19.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.917+0000 7f4b777fe640 1 -- 172.21.15.111:0/210752195 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 103131+0+0 (secure 0 0 0) 0x7f4b78003fb0 con 0x7f4b88102d80 2024-09-16T08:59:19.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.917+0000 7f4b777fe640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7007adb0 0x7f4b7007d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:19.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.917+0000 7f4b777fe640 1 -- 172.21.15.111:0/210752195 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f4b7809e520 con 0x7f4b88102d80 2024-09-16T08:59:19.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.917+0000 7f4b85d9b640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7007adb0 0x7f4b7007d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:19.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.918+0000 7f4b85d9b640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7007adb0 0x7f4b7007d270 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4b88107f30 tx=0x7f4b7c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:19.923 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:19.922+0000 7f4b777fe640 1 -- 172.21.15.111:0/210752195 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f4b780697e0 con 0x7f4b88102d80 2024-09-16T08:59:20.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.098+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b881998f0 con 0x7f4b88102d80 2024-09-16T08:59:20.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.099+0000 7f4b777fe640 1 -- 172.21.15.111:0/210752195 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+22846 (secure 0 0 0) 0x7f4b7806e690 con 0x7f4b88102d80 2024-09-16T08:59:20.101 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:59:20.101 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":62,"fsid":"f774c998-7408-11ef-bceb-c7b262605968","created":"2024-09-16T08:53:08.313205+0000","modified":"2024-09-16T08:59:08.163141+0000","last_up_change":"2024-09-16T08:58:47.983219+0000","last_in_change":"2024-09-16T08:58:25.803407+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T08:56:27.171375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-16T08:58:56.648022+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-16T08:58:57.705626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-16T08:58:59.332374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-16T08:59:01.279250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-16T08:59:03.342142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b1e3493f-2bf8-489d-b890-6cba7976e3c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6803","nonce":1090026984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6805","nonce":1090026984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6809","nonce":1090026984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6807","nonce":1090026984}]},"public_addr":"172.21.15.111:6803/1090026984","cluster_addr":"172.21.15.111:6805/1090026984","heartbeat_back_addr":"172.21.15.111:6809/1090026984","heartbeat_front_addr":"172.21.15.111:6807/1090026984","state":["exists","up"]},{"osd":1,"uuid":"5c855195-4c6d-47c7-9c63-1e1cc3712c1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6811","nonce":3642230577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6813","nonce":3642230577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6817","nonce":3642230577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6815","nonce":3642230577}]},"public_addr":"172.21.15.111:6811/3642230577","cluster_addr":"172.21.15.111:6813/3642230577","heartbeat_back_addr":"172.21.15.111:6817/3642230577","heartbeat_front_addr":"172.21.15.111:6815/3642230577","state":["exists","up"]},{"osd":2,"uuid":"7650515a-da82-4064-83a2-d4cb3665f1ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6819","nonce":3654696929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6821","nonce":3654696929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6825","nonce":3654696929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6823","nonce":3654696929}]},"public_addr":"172.21.15.111:6819/3654696929","cluster_addr":"172.21.15.111:6821/3654696929","heartbeat_back_addr":"172.21.15.111:6825/3654696929","heartbeat_front_addr":"172.21.15.111:6823/3654696929","state":["exists","up"]},{"osd":3,"uuid":"4e778b16-cf6a-405a-8ae3-899c7b7988b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6827","nonce":1538456457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6829","nonce":1538456457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6833","nonce":1538456457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6831","nonce":1538456457}]},"public_addr":"172.21.15.111:6827/1538456457","cluster_addr":"172.21.15.111:6829/1538456457","heartbeat_back_addr":"172.21.15.111:6833/1538456457","heartbeat_front_addr":"172.21.15.111:6831/1538456457","state":["exists","up"]},{"osd":4,"uuid":"2a15db99-3151-4039-81a5-d651e5b5b756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6801","nonce":3634478750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6803","nonce":3634478750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6807","nonce":3634478750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6805","nonce":3634478750}]},"public_addr":"172.21.15.114:6801/3634478750","cluster_addr":"172.21.15.114:6803/3634478750","heartbeat_back_addr":"172.21.15.114:6807/3634478750","heartbeat_front_addr":"172.21.15.114:6805/3634478750","state":["exists","up"]},{"osd":5,"uuid":"45f16b15-1806-477e-96d5-2f84ab4cc399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6809","nonce":101553972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6811","nonce":101553972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6815","nonce":101553972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6813","nonce":101553972}]},"public_addr":"172.21.15.114:6809/101553972","cluster_addr":"172.21.15.114:6811/101553972","heartbeat_back_addr":"172.21.15.114:6815/101553972","heartbeat_front_addr":"172.21.15.114:6813/101553972","state":["exists","up"]},{"osd":6,"uuid":"321fe560-27fd-4b30-a298-ae609af8eec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6817","nonce":3741048002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6819","nonce":3741048002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6823","nonce":3741048002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6821","nonce":3741048002}]},"public_addr":"172.21.15.114:6817/3741048002","cluster_addr":"172.21.15.114:6819/3741048002","heartbeat_back_addr":"172.21.15.114:6823/3741048002","heartbeat_front_addr":"172.21.15.114:6821/3741048002","state":["exists","up"]},{"osd":7,"uuid":"16273109-6547-46b3-b670-6173327b7677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6825","nonce":2543648078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6827","nonce":2543648078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6831","nonce":2543648078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6829","nonce":2543648078}]},"public_addr":"172.21.15.114:6825/2543648078","cluster_addr":"172.21.15.114:6827/2543648078","heartbeat_back_addr":"172.21.15.114:6831/2543648078","heartbeat_front_addr":"172.21.15.114:6829/2543648078","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:55:25.539345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:55:54.372473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:56:24.402273+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:56:53.591048+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:57:20.144327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:57:49.193663+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:58:17.683380+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:58:46.616167+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:0/1868356259":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/2123840083":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/2055672369":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/2883300306":"2024-09-17T08:53:39.765611+0000","172.21.15.111:6801/2366462893":"2024-09-17T08:54:06.869856+0000","172.21.15.111:6801/2026406736":"2024-09-17T08:53:39.765611+0000","172.21.15.111:6800/2026406736":"2024-09-17T08:53:39.765611+0000","172.21.15.111:0/3024901697":"2024-09-17T08:53:39.765611+0000","172.21.15.111:6800/2366462893":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/1333352500":"2024-09-17T08:53:39.765611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T08:59:20.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.101+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7007adb0 msgr2=0x7f4b7007d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.101+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7007adb0 0x7f4b7007d270 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4b88107f30 tx=0x7f4b7c005a20 comp rx=0 tx=0).stop 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.101+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b88102d80 msgr2=0x7f4b881987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.101+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b88102d80 0x7f4b881987c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800e9f0 tx=0x7f4b7800eeb0 comp rx=0 tx=0).stop 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.101+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 shutdown_connections 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.101+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f4b7007adb0 0x7f4b7007d270 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.101+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b88109110 0x7f4b8819e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.101+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4b88108790 0x7f4b88198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.101+0000 7f4b8cf92640 1 --2- 172.21.15.111:0/210752195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b88102d80 0x7f4b881987c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.102+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 >> 172.21.15.111:0/210752195 conn(0x7f4b880fe7d0 msgr2=0x7f4b88100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.102+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 shutdown_connections 2024-09-16T08:59:20.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:20.102+0000 7f4b8cf92640 1 -- 172.21.15.111:0/210752195 wait complete. 2024-09-16T08:59:20.113 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:20 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T08:59:20.826 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-16T08:59:20.827 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T08:59:21.075 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:21.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:20 smithi114 ceph-mon[33315]: pgmap v208: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 666 B/s wr, 27 op/s 2024-09-16T08:59:21.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/210752195' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T08:59:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:20 smithi111 ceph-mon[29803]: pgmap v208: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 666 B/s wr, 27 op/s 2024-09-16T08:59:21.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/210752195' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T08:59:21.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:20 smithi111 ceph-mon[40056]: pgmap v208: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 666 B/s wr, 27 op/s 2024-09-16T08:59:21.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/210752195' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T08:59:21.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.521+0000 7fc856c56640 1 -- 172.21.15.111:0/4253229847 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850108790 msgr2=0x7fc850108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:21.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.521+0000 7fc856c56640 1 --2- 172.21.15.111:0/4253229847 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850108790 0x7fc850108bd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc838009900 tx=0x7fc83802a410 comp rx=0 tx=0).stop 2024-09-16T08:59:21.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.521+0000 7fc856c56640 1 -- 172.21.15.111:0/4253229847 shutdown_connections 2024-09-16T08:59:21.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.521+0000 7fc856c56640 1 --2- 172.21.15.111:0/4253229847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc850109110 0x7fc85010f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:21.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.521+0000 7fc856c56640 1 --2- 172.21.15.111:0/4253229847 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850108790 0x7fc850108bd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.521+0000 7fc856c56640 1 --2- 172.21.15.111:0/4253229847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc850102d80 0x7fc850103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.521+0000 7fc856c56640 1 -- 172.21.15.111:0/4253229847 >> 172.21.15.111:0/4253229847 conn(0x7fc8500fe7d0 msgr2=0x7fc850100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.521+0000 7fc856c56640 1 -- 172.21.15.111:0/4253229847 shutdown_connections 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.521+0000 7fc856c56640 1 -- 172.21.15.111:0/4253229847 wait complete. 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc856c56640 1 Processor -- start 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc856c56640 1 -- start start 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc856c56640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850102d80 0x7fc8500754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc856c56640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc850108790 0x7fc8500759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc856c56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc850109110 0x7fc850075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc856c56640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501122f0 con 0x7fc850108790 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc856c56640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc850112170 con 0x7fc850109110 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc856c56640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fc850112470 con 0x7fc850102d80 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc8549f2640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850102d80 0x7fc8500754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc84ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc850108790 0x7fc8500759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc8549f2640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850102d80 0x7fc8500754a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:48390/0 (socket says 172.21.15.111:48390) 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc8549f2640 1 -- 172.21.15.111:0/2291089019 learned_addr learned my addr 172.21.15.111:0/2291089019 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.522+0000 7fc8551f3640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc850109110 0x7fc850075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc8549f2640 1 -- 172.21.15.111:0/2291089019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc850109110 msgr2=0x7fc850075f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc8549f2640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc850109110 0x7fc850075f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc8549f2640 1 -- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc850108790 msgr2=0x7fc8500759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc8549f2640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc850108790 0x7fc8500759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc8549f2640 1 -- 172.21.15.111:0/2291089019 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8501a6c20 con 0x7fc850102d80 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc8549f2640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850102d80 0x7fc8500754a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc84000c970 tx=0x7fc84000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc84dffb640 1 -- 172.21.15.111:0/2291089019 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840007bb0 con 0x7fc850102d80 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc84dffb640 1 -- 172.21.15.111:0/2291089019 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fc840007d50 con 0x7fc850102d80 2024-09-16T08:59:21.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8501a6f10 con 0x7fc850102d80 2024-09-16T08:59:21.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc84dffb640 1 -- 172.21.15.111:0/2291089019 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840004c90 con 0x7fc850102d80 2024-09-16T08:59:21.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.523+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8501a7370 con 0x7fc850102d80 2024-09-16T08:59:21.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.524+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc850103160 con 0x7fc850102d80 2024-09-16T08:59:21.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.525+0000 7fc84dffb640 1 -- 172.21.15.111:0/2291089019 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 16) ==== 103131+0+0 (secure 0 0 0) 0x7fc840004eb0 con 0x7fc850102d80 2024-09-16T08:59:21.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.525+0000 7fc84dffb640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fc83c07adb0 0x7fc83c07d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:21.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.525+0000 7fc84ffff640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fc83c07adb0 0x7fc83c07d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:21.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.525+0000 7fc84dffb640 1 -- 172.21.15.111:0/2291089019 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7fc84009dfe0 con 0x7fc850102d80 2024-09-16T08:59:21.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.525+0000 7fc84ffff640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fc83c07adb0 0x7fc83c07d270 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc838005ab0 tx=0x7fc838005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:21.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.529+0000 7fc84dffb640 1 -- 172.21.15.111:0/2291089019 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fc840014030 con 0x7fc850102d80 2024-09-16T08:59:21.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.706+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc85010aef0 con 0x7fc850102d80 2024-09-16T08:59:21.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.708+0000 7fc84dffb640 1 -- 172.21.15.111:0/2291089019 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+22846 (secure 0 0 0) 0x7fc8400692a0 con 0x7fc850102d80 2024-09-16T08:59:21.710 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:59:21.710 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":62,"fsid":"f774c998-7408-11ef-bceb-c7b262605968","created":"2024-09-16T08:53:08.313205+0000","modified":"2024-09-16T08:59:08.163141+0000","last_up_change":"2024-09-16T08:58:47.983219+0000","last_in_change":"2024-09-16T08:58:25.803407+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T08:56:27.171375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-16T08:58:56.648022+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-16T08:58:57.705626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-16T08:58:59.332374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-16T08:59:01.279250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-16T08:59:03.342142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b1e3493f-2bf8-489d-b890-6cba7976e3c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6803","nonce":1090026984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6805","nonce":1090026984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6809","nonce":1090026984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":1090026984},{"type":"v1","addr":"172.21.15.111:6807","nonce":1090026984}]},"public_addr":"172.21.15.111:6803/1090026984","cluster_addr":"172.21.15.111:6805/1090026984","heartbeat_back_addr":"172.21.15.111:6809/1090026984","heartbeat_front_addr":"172.21.15.111:6807/1090026984","state":["exists","up"]},{"osd":1,"uuid":"5c855195-4c6d-47c7-9c63-1e1cc3712c1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6811","nonce":3642230577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6813","nonce":3642230577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6817","nonce":3642230577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":3642230577},{"type":"v1","addr":"172.21.15.111:6815","nonce":3642230577}]},"public_addr":"172.21.15.111:6811/3642230577","cluster_addr":"172.21.15.111:6813/3642230577","heartbeat_back_addr":"172.21.15.111:6817/3642230577","heartbeat_front_addr":"172.21.15.111:6815/3642230577","state":["exists","up"]},{"osd":2,"uuid":"7650515a-da82-4064-83a2-d4cb3665f1ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6819","nonce":3654696929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6821","nonce":3654696929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6825","nonce":3654696929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3654696929},{"type":"v1","addr":"172.21.15.111:6823","nonce":3654696929}]},"public_addr":"172.21.15.111:6819/3654696929","cluster_addr":"172.21.15.111:6821/3654696929","heartbeat_back_addr":"172.21.15.111:6825/3654696929","heartbeat_front_addr":"172.21.15.111:6823/3654696929","state":["exists","up"]},{"osd":3,"uuid":"4e778b16-cf6a-405a-8ae3-899c7b7988b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6827","nonce":1538456457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6829","nonce":1538456457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6833","nonce":1538456457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1538456457},{"type":"v1","addr":"172.21.15.111:6831","nonce":1538456457}]},"public_addr":"172.21.15.111:6827/1538456457","cluster_addr":"172.21.15.111:6829/1538456457","heartbeat_back_addr":"172.21.15.111:6833/1538456457","heartbeat_front_addr":"172.21.15.111:6831/1538456457","state":["exists","up"]},{"osd":4,"uuid":"2a15db99-3151-4039-81a5-d651e5b5b756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6801","nonce":3634478750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6803","nonce":3634478750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6807","nonce":3634478750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3634478750},{"type":"v1","addr":"172.21.15.114:6805","nonce":3634478750}]},"public_addr":"172.21.15.114:6801/3634478750","cluster_addr":"172.21.15.114:6803/3634478750","heartbeat_back_addr":"172.21.15.114:6807/3634478750","heartbeat_front_addr":"172.21.15.114:6805/3634478750","state":["exists","up"]},{"osd":5,"uuid":"45f16b15-1806-477e-96d5-2f84ab4cc399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6809","nonce":101553972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6811","nonce":101553972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6815","nonce":101553972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":101553972},{"type":"v1","addr":"172.21.15.114:6813","nonce":101553972}]},"public_addr":"172.21.15.114:6809/101553972","cluster_addr":"172.21.15.114:6811/101553972","heartbeat_back_addr":"172.21.15.114:6815/101553972","heartbeat_front_addr":"172.21.15.114:6813/101553972","state":["exists","up"]},{"osd":6,"uuid":"321fe560-27fd-4b30-a298-ae609af8eec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6817","nonce":3741048002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6819","nonce":3741048002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6823","nonce":3741048002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3741048002},{"type":"v1","addr":"172.21.15.114:6821","nonce":3741048002}]},"public_addr":"172.21.15.114:6817/3741048002","cluster_addr":"172.21.15.114:6819/3741048002","heartbeat_back_addr":"172.21.15.114:6823/3741048002","heartbeat_front_addr":"172.21.15.114:6821/3741048002","state":["exists","up"]},{"osd":7,"uuid":"16273109-6547-46b3-b670-6173327b7677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6825","nonce":2543648078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6827","nonce":2543648078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6831","nonce":2543648078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2543648078},{"type":"v1","addr":"172.21.15.114:6829","nonce":2543648078}]},"public_addr":"172.21.15.114:6825/2543648078","cluster_addr":"172.21.15.114:6827/2543648078","heartbeat_back_addr":"172.21.15.114:6831/2543648078","heartbeat_front_addr":"172.21.15.114:6829/2543648078","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:55:25.539345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:55:54.372473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:56:24.402273+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:56:53.591048+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:57:20.144327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:57:49.193663+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:58:17.683380+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T08:58:46.616167+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:0/1868356259":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/2123840083":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/2055672369":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/2883300306":"2024-09-17T08:53:39.765611+0000","172.21.15.111:6801/2366462893":"2024-09-17T08:54:06.869856+0000","172.21.15.111:6801/2026406736":"2024-09-17T08:53:39.765611+0000","172.21.15.111:6800/2026406736":"2024-09-17T08:53:39.765611+0000","172.21.15.111:0/3024901697":"2024-09-17T08:53:39.765611+0000","172.21.15.111:6800/2366462893":"2024-09-17T08:54:06.869856+0000","172.21.15.111:0/1333352500":"2024-09-17T08:53:39.765611+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T08:59:21.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fc83c07adb0 msgr2=0x7fc83c07d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:21.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fc83c07adb0 0x7fc83c07d270 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc838005ab0 tx=0x7fc838005a20 comp rx=0 tx=0).stop 2024-09-16T08:59:21.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850102d80 msgr2=0x7fc8500754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:21.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850102d80 0x7fc8500754a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc84000c970 tx=0x7fc84000ce30 comp rx=0 tx=0).stop 2024-09-16T08:59:21.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 shutdown_connections 2024-09-16T08:59:21.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fc83c07adb0 0x7fc83c07d270 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:21.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc850109110 0x7fc850075f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:21.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc850108790 0x7fc8500759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:21.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 --2- 172.21.15.111:0/2291089019 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc850102d80 0x7fc8500754a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:21.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.710+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 >> 172.21.15.111:0/2291089019 conn(0x7fc8500fe7d0 msgr2=0x7fc850100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:21.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.711+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 shutdown_connections 2024-09-16T08:59:21.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:21.711+0000 7fc856c56640 1 -- 172.21.15.111:0/2291089019 wait complete. 2024-09-16T08:59:22.012 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:21 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T08:59:22.012 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2291089019' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T08:59:22.012 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:21 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T08:59:22.012 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2291089019' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T08:59:22.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:21 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T08:59:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2291089019' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-16T08:59:22.164 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:22 smithi114 systemd[1]: Starting Ceph prometheus.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T08:59:22.407 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-16T08:59:22.408 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-16T08:59:22.408 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-16T08:59:22.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-16T08:59:22.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-16T08:59:22.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-16T08:59:22.410 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-16T08:59:22.410 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-16T08:59:22.663 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:22 smithi114 podman[61524]: 2024-09-16 08:59:22.170056072 +0000 UTC m=+0.017346330 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-16T08:59:22.946 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:22.947 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:22.947 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:22.948 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:22.948 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:22.950 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:22.950 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:22.975 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:23.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:22 smithi114 ceph-mon[33315]: pgmap v209: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 597 B/s wr, 24 op/s 2024-09-16T08:59:23.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:22 smithi111 ceph-mon[29803]: pgmap v209: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 597 B/s wr, 24 op/s 2024-09-16T08:59:23.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:22 smithi111 ceph-mon[40056]: pgmap v209: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 597 B/s wr, 24 op/s 2024-09-16T08:59:23.663 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:23 smithi114 podman[61524]: 2024-09-16 08:59:23.241485216 +0000 UTC m=+1.088775463 container create 531f3e0837669b097cbee4d3b6198dda6398510c83d910fe152ab3a699c80b2a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T08:59:24.151 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:23 smithi111 ceph-mon[29803]: pgmap v210: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-16T08:59:24.152 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:23 smithi111 ceph-mon[40056]: pgmap v210: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-16T08:59:24.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:23 smithi114 ceph-mon[33315]: pgmap v210: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-16T08:59:24.164 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 podman[61524]: 2024-09-16 08:59:24.044086466 +0000 UTC m=+1.891376721 container init 531f3e0837669b097cbee4d3b6198dda6398510c83d910fe152ab3a699c80b2a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T08:59:24.164 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 podman[61524]: 2024-09-16 08:59:24.049213895 +0000 UTC m=+1.896504143 container start 531f3e0837669b097cbee4d3b6198dda6398510c83d910fe152ab3a699c80b2a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T08:59:24.164 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.069Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-16T08:59:24.164 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.069Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-16T08:59:24.164 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.069Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi114 (none))" 2024-09-16T08:59:24.164 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.069Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-16T08:59:24.164 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.069Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-16T08:59:24.165 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.070Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-16T08:59:24.165 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.071Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-16T08:59:24.165 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.072Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-16T08:59:24.165 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.072Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-16T08:59:24.165 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.074Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-16T08:59:24.165 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.074Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.798µs 2024-09-16T08:59:24.168 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.074Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-16T08:59:24.168 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.075Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-16T08:59:24.168 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.075Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.453µs wal_replay_duration=171.908µs wbl_replay_duration=108ns total_replay_duration=228.478µs 2024-09-16T08:59:24.168 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.076Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-16T08:59:24.168 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.076Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-16T08:59:24.168 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.076Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-16T08:59:24.169 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.091Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.36107ms db_storage=772ns remote_storage=1.06µs web_handler=256ns query_engine=542ns scrape=416.16µs scrape_sd=109.44µs notify=849ns notify_sd=905ns rules=14.528793ms tracing=7.309µs 2024-09-16T08:59:24.169 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.091Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-16T08:59:24.169 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T08:59:24.092Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-16T08:59:24.169 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 bash[61524]: 531f3e0837669b097cbee4d3b6198dda6398510c83d910fe152ab3a699c80b2a 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.291+0000 7f8394411640 1 -- 172.21.15.111:0/233100682 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c108790 msgr2=0x7f838c108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.291+0000 7f8394411640 1 --2- 172.21.15.111:0/233100682 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c108790 0x7f838c108bd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f837c009900 tx=0x7f837c02a410 comp rx=0 tx=0).stop 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.291+0000 7f8394411640 1 -- 172.21.15.111:0/233100682 shutdown_connections 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.291+0000 7f8394411640 1 --2- 172.21.15.111:0/233100682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f838c109110 0x7f838c10f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.291+0000 7f8394411640 1 --2- 172.21.15.111:0/233100682 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c108790 0x7f838c108bd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.291+0000 7f8394411640 1 --2- 172.21.15.111:0/233100682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c102d80 0x7f838c103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.291+0000 7f8394411640 1 -- 172.21.15.111:0/233100682 >> 172.21.15.111:0/233100682 conn(0x7f838c0fe7d0 msgr2=0x7f838c100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.291+0000 7f8394411640 1 -- 172.21.15.111:0/233100682 shutdown_connections 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.291+0000 7f8394411640 1 -- 172.21.15.111:0/233100682 wait complete. 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f8394411640 1 Processor -- start 2024-09-16T08:59:24.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f8394411640 1 -- start start 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f8394411640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c102d80 0x7f838c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f8394411640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c108790 0x7f838c198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f8394411640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f838c109110 0x7f838c19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f8394411640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f838c112280 con 0x7f838c102d80 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f8394411640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f838c112100 con 0x7f838c109110 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f8394411640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f838c112400 con 0x7f838c108790 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83921ad640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c102d80 0x7f838c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83919ac640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c108790 0x7f838c198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83921ad640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c102d80 0x7f838c1987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45144/0 (socket says 172.21.15.111:45144) 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83921ad640 1 -- 172.21.15.111:0/2370011905 learned_addr learned my addr 172.21.15.111:0/2370011905 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83929ae640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f838c109110 0x7f838c19e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83919ac640 1 -- 172.21.15.111:0/2370011905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f838c109110 msgr2=0x7f838c19e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83919ac640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f838c109110 0x7f838c19e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83919ac640 1 -- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c102d80 msgr2=0x7f838c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83919ac640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c102d80 0x7f838c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83919ac640 1 -- 172.21.15.111:0/2370011905 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838c19e860 con 0x7f838c108790 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.292+0000 7f83921ad640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c102d80 0x7f838c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.293+0000 7f83919ac640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c108790 0x7f838c198cf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f837c02aa00 tx=0x7f837c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.293+0000 7f83837fe640 1 -- 172.21.15.111:0/2370011905 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c03f070 con 0x7f838c108790 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.293+0000 7f83837fe640 1 -- 172.21.15.111:0/2370011905 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f837c03bab0 con 0x7f838c108790 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.293+0000 7f83837fe640 1 -- 172.21.15.111:0/2370011905 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c035030 con 0x7f838c108790 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.293+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f838c19eaf0 con 0x7f838c108790 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.293+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f838c075380 con 0x7f838c108790 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.294+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f838c063860 con 0x7f838c108790 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.294+0000 7f83837fe640 1 -- 172.21.15.111:0/2370011905 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 16) ==== 103131+0+0 (secure 0 0 0) 0x7f837c042070 con 0x7f838c108790 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.295+0000 7f83837fe640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f837807adb0 0x7f837807d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.295+0000 7f83837fe640 1 -- 172.21.15.111:0/2370011905 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f837c0be900 con 0x7f838c108790 2024-09-16T08:59:24.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.295+0000 7f83921ad640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f837807adb0 0x7f837807d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.295+0000 7f83837fe640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] conn(0x7f8378084900 0x7f8378086d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.295+0000 7f83837fe640 1 -- 172.21.15.111:0/2370011905 --> [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f837c032e50 con 0x7f8378084900 2024-09-16T08:59:24.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.295+0000 7f83837fe640 1 -- 172.21.15.111:0/2370011905 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f837c0c32c0 con 0x7f838c108790 2024-09-16T08:59:24.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.295+0000 7f83929ae640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] conn(0x7f8378084900 0x7f8378086d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.295+0000 7f83921ad640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f837807adb0 0x7f837807d270 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f83880095f0 tx=0x7f8388012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:24.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.295+0000 7f83929ae640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] conn(0x7f8378084900 0x7f8378086d60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:24.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.296+0000 7f83837fe640 1 -- 172.21.15.111:0/2370011905 <== osd.3 v2:172.21.15.111:6826/1538456457 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f837c032e50 con 0x7f8378084900 2024-09-16T08:59:24.314 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.309+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 --> [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f838c063c60 con 0x7f8378084900 2024-09-16T08:59:24.314 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f83837fe640 1 -- 172.21.15.111:0/2370011905 <== osd.3 v2:172.21.15.111:6826/1538456457 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f838c063c60 con 0x7f8378084900 2024-09-16T08:59:24.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] conn(0x7f8378084900 msgr2=0x7f8378086d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6826/1538456457,v1:172.21.15.111:6827/1538456457] conn(0x7f8378084900 0x7f8378086d60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f837807adb0 msgr2=0x7f837807d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f837807adb0 0x7f837807d270 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f83880095f0 tx=0x7f8388012040 comp rx=0 tx=0).stop 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c108790 msgr2=0x7f838c198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 --2- 172.21.15.111:0/2370011905 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c108790 0x7f838c198cf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f837c02aa00 tx=0x7f837c003b90 comp rx=0 tx=0).stop 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f83929ae640 1 -- 172.21.15.111:0/2370011905 reap_dead start 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 shutdown_connections 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 >> 172.21.15.111:0/2370011905 conn(0x7f838c0fe7d0 msgr2=0x7f838c100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 shutdown_connections 2024-09-16T08:59:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.310+0000 7f8394411640 1 -- 172.21.15.111:0/2370011905 wait complete. 2024-09-16T08:59:24.663 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 08:59:24 smithi114 systemd[1]: Started Ceph prometheus.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:59:24.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.657+0000 7fd47091d640 1 -- 172.21.15.111:0/3107575001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd46c1017f0 msgr2=0x7fd46c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.657+0000 7fd47091d640 1 --2- 172.21.15.111:0/3107575001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd46c1017f0 0x7fd46c101bd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009860 tx=0x7fd45c02a340 comp rx=0 tx=0).stop 2024-09-16T08:59:24.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.657+0000 7fd47091d640 1 -- 172.21.15.111:0/3107575001 shutdown_connections 2024-09-16T08:59:24.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.657+0000 7fd47091d640 1 --2- 172.21.15.111:0/3107575001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd46c1077a0 0x7fd46c10dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.657+0000 7fd47091d640 1 --2- 172.21.15.111:0/3107575001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c106e20 0x7fd46c107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.657+0000 7fd47091d640 1 --2- 172.21.15.111:0/3107575001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd46c1017f0 0x7fd46c101bd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.657+0000 7fd47091d640 1 -- 172.21.15.111:0/3107575001 >> 172.21.15.111:0/3107575001 conn(0x7fd46c076f50 msgr2=0x7fd46c077360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:24.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.657+0000 7fd47091d640 1 -- 172.21.15.111:0/3107575001 shutdown_connections 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.657+0000 7fd47091d640 1 -- 172.21.15.111:0/3107575001 wait complete. 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd47091d640 1 Processor -- start 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd47091d640 1 -- start start 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd47091d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd46c1017f0 0x7fd46c0fe690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd47091d640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd46c106e20 0x7fd46c0febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd47091d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c1077a0 0x7fd46c0ff110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd47091d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c110830 con 0x7fd46c1077a0 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd47091d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c1106b0 con 0x7fd46c1017f0 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd47091d640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fd46c1109b0 con 0x7fd46c106e20 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd469d9b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd46c106e20 0x7fd46c0febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd46ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c1077a0 0x7fd46c0ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd469d9b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd46c106e20 0x7fd46c0febd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:48424/0 (socket says 172.21.15.111:48424) 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd46ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c1077a0 0x7fd46c0ff110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45174/0 (socket says 172.21.15.111:45174) 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd469d9b640 1 -- 172.21.15.111:0/1832693510 learned_addr learned my addr 172.21.15.111:0/1832693510 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:24.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.658+0000 7fd46a59c640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd46c1017f0 0x7fd46c0fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd46ad9d640 1 -- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd46c106e20 msgr2=0x7fd46c0febd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd46ad9d640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd46c106e20 0x7fd46c0febd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd46ad9d640 1 -- 172.21.15.111:0/1832693510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd46c1017f0 msgr2=0x7fd46c0fe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd46ad9d640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd46c1017f0 0x7fd46c0fe690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd46ad9d640 1 -- 172.21.15.111:0/1832693510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46c1a6cc0 con 0x7fd46c1077a0 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd469d9b640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd46c106e20 0x7fd46c0febd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd46a59c640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd46c1017f0 0x7fd46c0fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd46ad9d640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c1077a0 0x7fd46c0ff110 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd46400e940 tx=0x7fd46400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd45b7fe640 1 -- 172.21.15.111:0/1832693510 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd464004d20 con 0x7fd46c1077a0 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd45b7fe640 1 -- 172.21.15.111:0/1832693510 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fd464005020 con 0x7fd46c1077a0 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd46c1a6fb0 con 0x7fd46c1077a0 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd45b7fe640 1 -- 172.21.15.111:0/1832693510 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4640116d0 con 0x7fd46c1077a0 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.659+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd46c1a7410 con 0x7fd46c1077a0 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.660+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd46c063860 con 0x7fd46c1077a0 2024-09-16T08:59:24.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.661+0000 7fd45b7fe640 1 -- 172.21.15.111:0/1832693510 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 103131+0+0 (secure 0 0 0) 0x7fd46400cb20 con 0x7fd46c1077a0 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.661+0000 7fd45b7fe640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd45407adb0 0x7fd45407d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.661+0000 7fd46a59c640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd45407adb0 0x7fd45407d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.661+0000 7fd45b7fe640 1 -- 172.21.15.111:0/1832693510 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7fd46409e840 con 0x7fd46c1077a0 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.662+0000 7fd45b7fe640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] conn(0x7fd454084880 0x7fd454086ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.662+0000 7fd45b7fe640 1 -- 172.21.15.111:0/1832693510 --> [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd46401ce50 con 0x7fd454084880 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.662+0000 7fd45b7fe640 1 -- 172.21.15.111:0/1832693510 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fd46409ebf0 con 0x7fd46c1077a0 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.662+0000 7fd469d9b640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] conn(0x7fd454084880 0x7fd454086ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.662+0000 7fd46a59c640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd45407adb0 0x7fd45407d270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd45c02a930 tx=0x7fd45c0058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.662+0000 7fd469d9b640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] conn(0x7fd454084880 0x7fd454086ce0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.663+0000 7fd45b7fe640 1 -- 172.21.15.111:0/1832693510 <== osd.2 v2:172.21.15.111:6818/3654696929 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd46401ce50 con 0x7fd454084880 2024-09-16T08:59:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.676+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 --> [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd46c063c60 con 0x7fd454084880 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.676+0000 7fd45b7fe640 1 -- 172.21.15.111:0/1832693510 <== osd.2 v2:172.21.15.111:6818/3654696929 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd46c063c60 con 0x7fd454084880 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.676+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] conn(0x7fd454084880 msgr2=0x7fd454086ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.676+0000 7fd47091d640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6818/3654696929,v1:172.21.15.111:6819/3654696929] conn(0x7fd454084880 0x7fd454086ce0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.677+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd45407adb0 msgr2=0x7fd45407d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.677+0000 7fd47091d640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fd45407adb0 0x7fd45407d270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd45c02a930 tx=0x7fd45c0058d0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.677+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c1077a0 msgr2=0x7fd46c0ff110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.677+0000 7fd47091d640 1 --2- 172.21.15.111:0/1832693510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd46c1077a0 0x7fd46c0ff110 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd46400e940 tx=0x7fd46400ee00 comp rx=0 tx=0).stop 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.677+0000 7fd46ad9d640 1 -- 172.21.15.111:0/1832693510 reap_dead start 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.677+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 shutdown_connections 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.677+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 >> 172.21.15.111:0/1832693510 conn(0x7fd46c076f50 msgr2=0x7fd46c106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.677+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 shutdown_connections 2024-09-16T08:59:24.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.677+0000 7fd47091d640 1 -- 172.21.15.111:0/1832693510 wait complete. 2024-09-16T08:59:24.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1311112312 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfe4103c90 msgr2=0x7fbfe4104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 --2- 172.21.15.111:0/1311112312 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfe4103c90 0x7fbfe4104070 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009900 tx=0x7fbfd402a410 comp rx=0 tx=0).stop 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1311112312 shutdown_connections 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 --2- 172.21.15.111:0/1311112312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfe410d0f0 0x7fbfe410f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 --2- 172.21.15.111:0/1311112312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe41045b0 0x7fbfe410caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 --2- 172.21.15.111:0/1311112312 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfe4103c90 0x7fbfe4104070 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1311112312 >> 172.21.15.111:0/1311112312 conn(0x7fbfe40fd6f0 msgr2=0x7fbfe40ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1311112312 shutdown_connections 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1311112312 wait complete. 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.767+0000 7fbfe9b19640 1 Processor -- start 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe9b19640 1 -- start start 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe9b19640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfe4103c90 0x7fbfe4101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe9b19640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfe41045b0 0x7fbfe4101e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe9b19640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe410d0f0 0x7fbfe41023c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe9b19640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe4112230 con 0x7fbfe41045b0 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe9b19640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe41120b0 con 0x7fbfe410d0f0 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe9b19640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe41123b0 con 0x7fbfe4103c90 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe37fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfe4103c90 0x7fbfe4101940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe2ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfe41045b0 0x7fbfe4101e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe37fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfe4103c90 0x7fbfe4101940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:48446/0 (socket says 172.21.15.111:48446) 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe37fe640 1 -- 172.21.15.111:0/1451310180 learned_addr learned my addr 172.21.15.111:0/1451310180 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe3fff640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe410d0f0 0x7fbfe41023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe37fe640 1 -- 172.21.15.111:0/1451310180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe410d0f0 msgr2=0x7fbfe41023c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe37fe640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe410d0f0 0x7fbfe41023c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe37fe640 1 -- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfe41045b0 msgr2=0x7fbfe4101e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe37fe640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfe41045b0 0x7fbfe4101e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe37fe640 1 -- 172.21.15.111:0/1451310180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe41a6c90 con 0x7fbfe4103c90 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe3fff640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe410d0f0 0x7fbfe41023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T08:59:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe2ffd640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfe41045b0 0x7fbfe4101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe37fe640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfe4103c90 0x7fbfe4101940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4005940 tx=0x7fbfd4032b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.768+0000 7fbfe0ff9640 1 -- 172.21.15.111:0/1451310180 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd403f030 con 0x7fbfe4103c90 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.769+0000 7fbfe0ff9640 1 -- 172.21.15.111:0/1451310180 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fbfd4003a40 con 0x7fbfe4103c90 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.769+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe41a6f20 con 0x7fbfe4103c90 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.769+0000 7fbfe0ff9640 1 -- 172.21.15.111:0/1451310180 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4035070 con 0x7fbfe4103c90 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.769+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe41a7370 con 0x7fbfe4103c90 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.770+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbfe41063f0 con 0x7fbfe4103c90 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.770+0000 7fbfe0ff9640 1 -- 172.21.15.111:0/1451310180 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 16) ==== 103131+0+0 (secure 0 0 0) 0x7fbfd4042030 con 0x7fbfe4103c90 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.771+0000 7fbfe0ff9640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbfcc07adb0 0x7fbfcc07d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.771+0000 7fbfe2ffd640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbfcc07adb0 0x7fbfcc07d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.771+0000 7fbfe0ff9640 1 -- 172.21.15.111:0/1451310180 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7fbfd40be790 con 0x7fbfe4103c90 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.771+0000 7fbfe0ff9640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] conn(0x7fbfcc084880 0x7fbfcc086ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.771+0000 7fbfe0ff9640 1 -- 172.21.15.111:0/1451310180 --> [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfd4040e70 con 0x7fbfcc084880 2024-09-16T08:59:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.771+0000 7fbfe0ff9640 1 -- 172.21.15.111:0/1451310180 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fbfd40c32c0 con 0x7fbfe4103c90 2024-09-16T08:59:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.771+0000 7fbfe3fff640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] conn(0x7fbfcc084880 0x7fbfcc086ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.771+0000 7fbfe2ffd640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbfcc07adb0 0x7fbfcc07d270 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4103430 tx=0x7fbfd800b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.771+0000 7fbfe3fff640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] conn(0x7fbfcc084880 0x7fbfcc086ce0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.772+0000 7fbfe0ff9640 1 -- 172.21.15.111:0/1451310180 <== osd.6 v2:172.21.15.114:6816/3741048002 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbfd4040e70 con 0x7fbfcc084880 2024-09-16T08:59:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 --> [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbfe406bb60 con 0x7fbfcc084880 2024-09-16T08:59:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe0ff9640 1 -- 172.21.15.111:0/1451310180 <== osd.6 v2:172.21.15.114:6816/3741048002 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbfe406bb60 con 0x7fbfcc084880 2024-09-16T08:59:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 >> [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] conn(0x7fbfcc084880 msgr2=0x7fbfcc086ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe9b19640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.114:6816/3741048002,v1:172.21.15.114:6817/3741048002] conn(0x7fbfcc084880 0x7fbfcc086ce0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbfcc07adb0 msgr2=0x7fbfcc07d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe9b19640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fbfcc07adb0 0x7fbfcc07d270 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4103430 tx=0x7fbfd800b2e0 comp rx=0 tx=0).stop 2024-09-16T08:59:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfe4103c90 msgr2=0x7fbfe4101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe9b19640 1 --2- 172.21.15.111:0/1451310180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfe4103c90 0x7fbfe4101940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4005940 tx=0x7fbfd4032b70 comp rx=0 tx=0).stop 2024-09-16T08:59:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe3fff640 1 -- 172.21.15.111:0/1451310180 reap_dead start 2024-09-16T08:59:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 shutdown_connections 2024-09-16T08:59:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.786+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 >> 172.21.15.111:0/1451310180 conn(0x7fbfe40fd6f0 msgr2=0x7fbfe4104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.787+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 shutdown_connections 2024-09-16T08:59:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:24.787+0000 7fbfe9b19640 1 -- 172.21.15.111:0/1451310180 wait complete. 2024-09-16T08:59:25.627 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:25.627 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:25.627 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:25.628 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-16T08:59:25.628 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:25.628 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:25.628 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:25.628 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-16T08:59:25.628 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ignoring --setuser ceph since I am not root 2024-09-16T08:59:25.628 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:25 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ignoring --setgroup ceph since I am not root 2024-09-16T08:59:25.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:25.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:25.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' 2024-09-16T08:59:25.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:25 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-16T08:59:25.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:59:25 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x[34799]: ignoring --setuser ceph since I am not root 2024-09-16T08:59:25.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:59:25 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x[34799]: ignoring --setgroup ceph since I am not root 2024-09-16T08:59:25.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:59:25 smithi114 ceph-mgr[34822]: -- 172.21.15.114:0/1490378698 <== mon.0 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc373d30e0 con 0x55cc373b0800 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.708+0000 7f7b13e86640 1 -- 172.21.15.111:0/255887349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b0c101c60 msgr2=0x7f7b0c102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.708+0000 7f7b13e86640 1 --2- 172.21.15.111:0/255887349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b0c101c60 0x7f7b0c102040 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009900 tx=0x7f7b0802a410 comp rx=0 tx=0).stop 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.708+0000 7f7b13e86640 1 -- 172.21.15.111:0/255887349 shutdown_connections 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.708+0000 7f7b13e86640 1 --2- 172.21.15.111:0/255887349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b0c107ff0 0x7f7b0c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.708+0000 7f7b13e86640 1 --2- 172.21.15.111:0/255887349 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b0c107670 0x7f7b0c107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.708+0000 7f7b13e86640 1 --2- 172.21.15.111:0/255887349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b0c101c60 0x7f7b0c102040 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.708+0000 7f7b13e86640 1 -- 172.21.15.111:0/255887349 >> 172.21.15.111:0/255887349 conn(0x7f7b0c0fd6d0 msgr2=0x7f7b0c0ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.708+0000 7f7b13e86640 1 -- 172.21.15.111:0/255887349 shutdown_connections 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.708+0000 7f7b13e86640 1 -- 172.21.15.111:0/255887349 wait complete. 2024-09-16T08:59:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b13e86640 1 Processor -- start 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b13e86640 1 -- start start 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b13e86640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b0c101c60 0x7f7b0c19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b13e86640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b0c107670 0x7f7b0c19afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b13e86640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b0c107ff0 0x7f7b0c1a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b13e86640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c111190 con 0x7f7b0c107670 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b13e86640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c111010 con 0x7f7b0c107ff0 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b13e86640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0c111310 con 0x7f7b0c101c60 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11c22640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b0c101c60 0x7f7b0c19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11c22640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b0c101c60 0x7f7b0c19aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:48472/0 (socket says 172.21.15.111:48472) 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11421640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b0c107670 0x7f7b0c19afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11c22640 1 -- 172.21.15.111:0/2594811812 learned_addr learned my addr 172.21.15.111:0/2594811812 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b12423640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b0c107ff0 0x7f7b0c1a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11421640 1 -- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b0c101c60 msgr2=0x7f7b0c19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11421640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b0c101c60 0x7f7b0c19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:25.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11421640 1 -- 172.21.15.111:0/2594811812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b0c107ff0 msgr2=0x7f7b0c1a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:25.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11421640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b0c107ff0 0x7f7b0c1a0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:25.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11421640 1 -- 172.21.15.111:0/2594811812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0c1a0b10 con 0x7f7b0c107670 2024-09-16T08:59:25.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.709+0000 7f7b11c22640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b0c101c60 0x7f7b0c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:25.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.710+0000 7f7b11421640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b0c107670 0x7f7b0c19afa0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00e9f0 tx=0x7f7afc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:25.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.710+0000 7f7b02ffd640 1 -- 172.21.15.111:0/2594811812 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc00cd80 con 0x7f7b0c107670 2024-09-16T08:59:25.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.710+0000 7f7b02ffd640 1 -- 172.21.15.111:0/2594811812 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f7afc004470 con 0x7f7b0c107670 2024-09-16T08:59:25.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.710+0000 7f7b02ffd640 1 -- 172.21.15.111:0/2594811812 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc003e10 con 0x7f7b0c107670 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.710+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0c1a0e00 con 0x7f7b0c107670 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.710+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0c1a74c0 con 0x7f7b0c107670 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.711+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b0c10ddb0 con 0x7f7b0c107670 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.711+0000 7f7b02ffd640 1 -- 172.21.15.111:0/2594811812 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) ==== 103145+0+0 (secure 0 0 0) 0x7f7afc004020 con 0x7f7b0c107670 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.712+0000 7f7b02ffd640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7af807ae00 0x7f7af807d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.712+0000 7f7b11c22640 1 -- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7af807ae00 msgr2=0x7f7af807d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3306957597 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.712+0000 7f7b02ffd640 1 -- 172.21.15.111:0/2594811812 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f7afc09e580 con 0x7f7b0c107670 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.712+0000 7f7b11c22640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7af807ae00 0x7f7af807d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.712+0000 7f7b02ffd640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] conn(0x7f7af80848d0 0x7f7af8086d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.712+0000 7f7b02ffd640 1 -- 172.21.15.111:0/2594811812 --> [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7afc015e80 con 0x7f7af80848d0 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.712+0000 7f7b02ffd640 1 -- 172.21.15.111:0/2594811812 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f7afc0a5050 con 0x7f7b0c107670 2024-09-16T08:59:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.712+0000 7f7b12423640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] conn(0x7f7af80848d0 0x7f7af8086d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.713+0000 7f7b12423640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] conn(0x7f7af80848d0 0x7f7af8086d30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.713+0000 7f7b02ffd640 1 -- 172.21.15.111:0/2594811812 <== osd.5 v2:172.21.15.114:6808/101553972 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7afc015e80 con 0x7f7af80848d0 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.726+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 --> [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b0c102f10 con 0x7f7af80848d0 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b02ffd640 1 -- 172.21.15.111:0/2594811812 <== osd.5 v2:172.21.15.114:6808/101553972 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7b0c102f10 con 0x7f7af80848d0 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 >> [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] conn(0x7f7af80848d0 msgr2=0x7f7af8086d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.114:6808/101553972,v1:172.21.15.114:6809/101553972] conn(0x7f7af80848d0 0x7f7af8086d30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7af807ae00 msgr2=0x7f7af807d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f7af807ae00 0x7f7af807d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b0c107670 msgr2=0x7f7b0c19afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 --2- 172.21.15.111:0/2594811812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b0c107670 0x7f7b0c19afa0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00e9f0 tx=0x7f7afc00eeb0 comp rx=0 tx=0).stop 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b12423640 1 -- 172.21.15.111:0/2594811812 reap_dead start 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 shutdown_connections 2024-09-16T08:59:25.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 >> 172.21.15.111:0/2594811812 conn(0x7f7b0c0fd6d0 msgr2=0x7f7b0c06b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:25.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 shutdown_connections 2024-09-16T08:59:25.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:25.727+0000 7f7b13e86640 1 -- 172.21.15.111:0/2594811812 wait complete. 2024-09-16T08:59:26.415 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:26 smithi111 ceph-mon[29803]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-16T08:59:26.416 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:26 smithi111 ceph-mon[29803]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-16T08:59:26.416 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:26 smithi111 ceph-mon[40056]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-16T08:59:26.416 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:26 smithi111 ceph-mon[40056]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-16T08:59:26.445 INFO:teuthology.orchestra.run.smithi111.stdout:184683593744 2024-09-16T08:59:26.445 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-16T08:59:26.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:26 smithi114 ceph-mon[33315]: from='mgr.14152 172.21.15.111:0/973747319' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-16T08:59:26.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:26 smithi114 ceph-mon[33315]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.070+0000 7ff002823640 1 -- 172.21.15.111:0/31659750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feffc107ff0 msgr2=0x7feffc10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.070+0000 7ff002823640 1 --2- 172.21.15.111:0/31659750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feffc107ff0 0x7feffc10e7f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feff4009ca0 tx=0x7feff402a8c0 comp rx=0 tx=0).stop 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.070+0000 7ff002823640 1 -- 172.21.15.111:0/31659750 shutdown_connections 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.070+0000 7ff002823640 1 --2- 172.21.15.111:0/31659750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feffc107ff0 0x7feffc10e7f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.070+0000 7ff002823640 1 --2- 172.21.15.111:0/31659750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feffc107670 0x7feffc107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.070+0000 7ff002823640 1 --2- 172.21.15.111:0/31659750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feffc101c60 0x7feffc102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.070+0000 7ff002823640 1 -- 172.21.15.111:0/31659750 >> 172.21.15.111:0/31659750 conn(0x7feffc0fd6d0 msgr2=0x7feffc0ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.070+0000 7ff002823640 1 -- 172.21.15.111:0/31659750 shutdown_connections 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.070+0000 7ff002823640 1 -- 172.21.15.111:0/31659750 wait complete. 2024-09-16T08:59:27.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff002823640 1 Processor -- start 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff002823640 1 -- start start 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff002823640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feffc101c60 0x7feffc197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff002823640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feffc107670 0x7feffc197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff002823640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feffc107ff0 0x7feffc19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff002823640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feffc111140 con 0x7feffc107ff0 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff002823640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feffc110fc0 con 0x7feffc101c60 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff002823640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7feffc1112c0 con 0x7feffc107670 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7feffb7fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feffc107670 0x7feffc197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff000dc0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feffc107ff0 0x7feffc19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7feffb7fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feffc107670 0x7feffc197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:48498/0 (socket says 172.21.15.111:48498) 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7feffb7fe640 1 -- 172.21.15.111:0/141158976 learned_addr learned my addr 172.21.15.111:0/141158976 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.071+0000 7ff000dc0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feffc107ff0 0x7feffc19cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45250/0 (socket says 172.21.15.111:45250) 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7feffbfff640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feffc101c60 0x7feffc197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:27.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7ff000dc0640 1 -- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feffc107670 msgr2=0x7feffc197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:27.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7ff000dc0640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feffc107670 0x7feffc197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:27.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7ff000dc0640 1 -- 172.21.15.111:0/141158976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feffc101c60 msgr2=0x7feffc197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:27.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7ff000dc0640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feffc101c60 0x7feffc197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:27.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7ff000dc0640 1 -- 172.21.15.111:0/141158976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc19d700 con 0x7feffc107ff0 2024-09-16T08:59:27.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7feffb7fe640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feffc107670 0x7feffc197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:27.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7feffbfff640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feffc101c60 0x7feffc197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:27.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7ff000dc0640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feffc107ff0 0x7feffc19cf30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7feff402ade0 tx=0x7feff4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:27.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7feff97fa640 1 -- 172.21.15.111:0/141158976 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4004040 con 0x7feffc107ff0 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feffc19d990 con 0x7feffc107ff0 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7feff97fa640 1 -- 172.21.15.111:0/141158976 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7feff4040070 con 0x7feffc107ff0 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7feff97fa640 1 -- 172.21.15.111:0/141158976 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4030ab0 con 0x7feffc107ff0 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.072+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feffc06aa80 con 0x7feffc107ff0 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.073+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feffc06b070 con 0x7feffc107ff0 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.074+0000 7feff97fa640 1 -- 172.21.15.111:0/141158976 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) ==== 103145+0+0 (secure 0 0 0) 0x7feff40046c0 con 0x7feffc107ff0 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.074+0000 7feff97fa640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fefe407ae00 0x7fefe407d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.075+0000 7feffbfff640 1 -- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fefe407ae00 msgr2=0x7fefe407d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3306957597 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.075+0000 7feff97fa640 1 -- 172.21.15.111:0/141158976 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7feff40bf620 con 0x7feffc107ff0 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.075+0000 7feffbfff640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fefe407ae00 0x7fefe407d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:59:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.075+0000 7feff97fa640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] conn(0x7fefe40848d0 0x7fefe4086d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.075+0000 7feff97fa640 1 -- 172.21.15.111:0/141158976 --> [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feff4039e50 con 0x7fefe40848d0 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.075+0000 7feff97fa640 1 -- 172.21.15.111:0/141158976 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7feff40bfb00 con 0x7feffc107ff0 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.075+0000 7feffb7fe640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] conn(0x7fefe40848d0 0x7fefe4086d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.075+0000 7feffb7fe640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] conn(0x7fefe40848d0 0x7fefe4086d30 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.076+0000 7feff97fa640 1 -- 172.21.15.111:0/141158976 <== osd.1 v2:172.21.15.111:6810/3642230577 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feff4039e50 con 0x7fefe40848d0 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.090+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 --> [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feffc06b470 con 0x7fefe40848d0 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7feff97fa640 1 -- 172.21.15.111:0/141158976 <== osd.1 v2:172.21.15.111:6810/3642230577 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7feffc06b470 con 0x7fefe40848d0 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] conn(0x7fefe40848d0 msgr2=0x7fefe4086d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6810/3642230577,v1:172.21.15.111:6811/3642230577] conn(0x7fefe40848d0 0x7fefe4086d30 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fefe407ae00 msgr2=0x7fefe407d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7fefe407ae00 0x7fefe407d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feffc107ff0 msgr2=0x7feffc19cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 --2- 172.21.15.111:0/141158976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feffc107ff0 0x7feffc19cf30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7feff402ade0 tx=0x7feff4003ba0 comp rx=0 tx=0).stop 2024-09-16T08:59:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff000dc0640 1 -- 172.21.15.111:0/141158976 reap_dead start 2024-09-16T08:59:27.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 shutdown_connections 2024-09-16T08:59:27.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 >> 172.21.15.111:0/141158976 conn(0x7feffc0fd6d0 msgr2=0x7feffc0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:27.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 shutdown_connections 2024-09-16T08:59:27.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:27.091+0000 7ff002823640 1 -- 172.21.15.111:0/141158976 wait complete. 2024-09-16T08:59:27.901 INFO:teuthology.orchestra.run.smithi111.stdout:77309411366 2024-09-16T08:59:27.902 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-16T08:59:28.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.128+0000 7f5c991d8640 1 -- 172.21.15.111:0/3865467153 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c94101c60 msgr2=0x7f5c94102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.128+0000 7f5c991d8640 1 --2- 172.21.15.111:0/3865467153 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c94101c60 0x7f5c94102040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009860 tx=0x7f5c8c02a340 comp rx=0 tx=0).stop 2024-09-16T08:59:28.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.128+0000 7f5c991d8640 1 -- 172.21.15.111:0/3865467153 shutdown_connections 2024-09-16T08:59:28.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.128+0000 7f5c991d8640 1 --2- 172.21.15.111:0/3865467153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c94107ff0 0x7f5c9410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.128+0000 7f5c991d8640 1 --2- 172.21.15.111:0/3865467153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94107670 0x7f5c94107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.128+0000 7f5c991d8640 1 --2- 172.21.15.111:0/3865467153 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c94101c60 0x7f5c94102040 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.128+0000 7f5c991d8640 1 -- 172.21.15.111:0/3865467153 >> 172.21.15.111:0/3865467153 conn(0x7f5c940fd6d0 msgr2=0x7f5c940ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.128+0000 7f5c991d8640 1 -- 172.21.15.111:0/3865467153 shutdown_connections 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.128+0000 7f5c991d8640 1 -- 172.21.15.111:0/3865467153 wait complete. 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.129+0000 7f5c991d8640 1 Processor -- start 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.129+0000 7f5c991d8640 1 -- start start 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c991d8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94101c60 0x7f5c941995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c991d8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c94107670 0x7f5c94199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c991d8640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c94107ff0 0x7f5c9419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c991d8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94111180 con 0x7f5c94107670 2024-09-16T08:59:28.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c991d8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94111000 con 0x7f5c94101c60 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c991d8640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f5c94111300 con 0x7f5c94107ff0 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c9259c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c94107670 0x7f5c94199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c92d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94101c60 0x7f5c941995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c9259c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c94107670 0x7f5c94199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45278/0 (socket says 172.21.15.111:45278) 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c9259c640 1 -- 172.21.15.111:0/3781342856 learned_addr learned my addr 172.21.15.111:0/3781342856 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c92d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94101c60 0x7f5c941995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.111:43580/0 (socket says 172.21.15.111:43580) 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c9359e640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c94107ff0 0x7f5c9419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c92d9d640 1 -- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c94107ff0 msgr2=0x7f5c9419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c92d9d640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c94107ff0 0x7f5c9419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c92d9d640 1 -- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c94107670 msgr2=0x7f5c94199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c92d9d640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c94107670 0x7f5c94199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c92d9d640 1 -- 172.21.15.111:0/3781342856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c941a49b0 con 0x7f5c94101c60 2024-09-16T08:59:28.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c9359e640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c94107ff0 0x7f5c9419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:28.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c9259c640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c94107670 0x7f5c94199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:28.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.130+0000 7f5c92d9d640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94101c60 0x7f5c941995d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c02a860 tx=0x7f5c8c033d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:28.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.131+0000 7f5c7bfff640 1 -- 172.21.15.111:0/3781342856 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c037040 con 0x7f5c94101c60 2024-09-16T08:59:28.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.131+0000 7f5c7bfff640 1 -- 172.21.15.111:0/3781342856 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f5c8c0306b0 con 0x7f5c94101c60 2024-09-16T08:59:28.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.131+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c941a4c40 con 0x7f5c94101c60 2024-09-16T08:59:28.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.131+0000 7f5c7bfff640 1 -- 172.21.15.111:0/3781342856 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c030e60 con 0x7f5c94101c60 2024-09-16T08:59:28.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.131+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c941a5120 con 0x7f5c94101c60 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.131+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5c94063860 con 0x7f5c94101c60 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.132+0000 7f5c7bfff640 1 -- 172.21.15.111:0/3781342856 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 103145+0+0 (secure 0 0 0) 0x7f5c8c042020 con 0x7f5c94101c60 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.133+0000 7f5c7bfff640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c7c07ae00 0x7f5c7c07d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.133+0000 7f5c9259c640 1 -- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c7c07ae00 msgr2=0x7f5c7c07d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3306957597 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.133+0000 7f5c7bfff640 1 -- 172.21.15.111:0/3781342856 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f5c8c0bd6c0 con 0x7f5c94101c60 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.133+0000 7f5c9259c640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c7c07ae00 0x7f5c7c07d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.133+0000 7f5c7bfff640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] conn(0x7f5c7c0848d0 0x7f5c7c086d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.133+0000 7f5c7bfff640 1 -- 172.21.15.111:0/3781342856 --> [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c8c007e80 con 0x7f5c7c0848d0 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.133+0000 7f5c7bfff640 1 -- 172.21.15.111:0/3781342856 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f5c8c0bdb20 con 0x7f5c94101c60 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.133+0000 7f5c9359e640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] conn(0x7f5c7c0848d0 0x7f5c7c086d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.134+0000 7f5c9359e640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] conn(0x7f5c7c0848d0 0x7f5c7c086d30 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.134+0000 7f5c7bfff640 1 -- 172.21.15.111:0/3781342856 <== osd.7 v2:172.21.15.114:6824/2543648078 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c8c007e80 con 0x7f5c7c0848d0 2024-09-16T08:59:28.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.148+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 --> [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5c94063c60 con 0x7f5c7c0848d0 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.148+0000 7f5c7bfff640 1 -- 172.21.15.111:0/3781342856 <== osd.7 v2:172.21.15.114:6824/2543648078 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5c94063c60 con 0x7f5c7c0848d0 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.148+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 >> [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] conn(0x7f5c7c0848d0 msgr2=0x7f5c7c086d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.148+0000 7f5c991d8640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.114:6824/2543648078,v1:172.21.15.114:6825/2543648078] conn(0x7f5c7c0848d0 0x7f5c7c086d30 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.149+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c7c07ae00 msgr2=0x7f5c7c07d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.149+0000 7f5c991d8640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c7c07ae00 0x7f5c7c07d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.149+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94101c60 msgr2=0x7f5c941995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.149+0000 7f5c991d8640 1 --2- 172.21.15.111:0/3781342856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94101c60 0x7f5c941995d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c02a860 tx=0x7f5c8c033d00 comp rx=0 tx=0).stop 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.149+0000 7f5c9359e640 1 -- 172.21.15.111:0/3781342856 reap_dead start 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.149+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 shutdown_connections 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.149+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 >> 172.21.15.111:0/3781342856 conn(0x7f5c940fd6d0 msgr2=0x7f5c940ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.149+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 shutdown_connections 2024-09-16T08:59:28.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.149+0000 7f5c991d8640 1 -- 172.21.15.111:0/3781342856 wait complete. 2024-09-16T08:59:28.852 INFO:teuthology.orchestra.run.smithi111.stdout:55834574893 2024-09-16T08:59:28.852 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-16T08:59:28.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.377+0000 7f5c6504d640 1 -- 172.21.15.111:0/1116801775 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c60101c60 msgr2=0x7f5c60102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.377+0000 7f5c6504d640 1 --2- 172.21.15.111:0/1116801775 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c60101c60 0x7f5c60102040 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5c50009860 tx=0x7f5c5002a340 comp rx=0 tx=0).stop 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.378+0000 7f5c6504d640 1 -- 172.21.15.111:0/1116801775 shutdown_connections 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.378+0000 7f5c6504d640 1 --2- 172.21.15.111:0/1116801775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c60107ff0 0x7f5c6010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.378+0000 7f5c6504d640 1 --2- 172.21.15.111:0/1116801775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c60107670 0x7f5c60107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.378+0000 7f5c6504d640 1 --2- 172.21.15.111:0/1116801775 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c60101c60 0x7f5c60102040 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.378+0000 7f5c6504d640 1 -- 172.21.15.111:0/1116801775 >> 172.21.15.111:0/1116801775 conn(0x7f5c600fd6d0 msgr2=0x7f5c600ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.378+0000 7f5c6504d640 1 -- 172.21.15.111:0/1116801775 shutdown_connections 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.378+0000 7f5c6504d640 1 -- 172.21.15.111:0/1116801775 wait complete. 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.378+0000 7f5c6504d640 1 Processor -- start 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c6504d640 1 -- start start 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c6504d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c60101c60 0x7f5c60197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c6504d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c60107670 0x7f5c60197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c6504d640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c60107ff0 0x7f5c6019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c6504d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5c60111180 con 0x7f5c60101c60 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c6504d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5c60111000 con 0x7f5c60107670 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c6504d640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f5c60111300 con 0x7f5c60107ff0 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c5ed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c60101c60 0x7f5c60197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c5f59e640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c60107ff0 0x7f5c6019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c5ed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c60101c60 0x7f5c60197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45298/0 (socket says 172.21.15.111:45298) 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c5f59e640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c60107ff0 0x7f5c6019cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:48534/0 (socket says 172.21.15.111:48534) 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c5ed9d640 1 -- 172.21.15.111:0/3779665792 learned_addr learned my addr 172.21.15.111:0/3779665792 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:28.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.379+0000 7f5c5e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c60107670 0x7f5c60197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c5e59c640 1 -- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c60107ff0 msgr2=0x7f5c6019cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c5e59c640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c60107ff0 0x7f5c6019cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c5e59c640 1 -- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c60101c60 msgr2=0x7f5c60197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c5e59c640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c60101c60 0x7f5c60197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c5e59c640 1 -- 172.21.15.111:0/3779665792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6019d740 con 0x7f5c60107670 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c5f59e640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c60107ff0 0x7f5c6019cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c5ed9d640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c60101c60 0x7f5c60197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c5e59c640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c60107670 0x7f5c60197bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5c54005fd0 tx=0x7f5c5400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c47fff640 1 -- 172.21.15.111:0/3779665792 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5400cd00 con 0x7f5c60107670 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c47fff640 1 -- 172.21.15.111:0/3779665792 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f5c54004470 con 0x7f5c60107670 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6019da30 con 0x7f5c60107670 2024-09-16T08:59:28.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c47fff640 1 -- 172.21.15.111:0/3779665792 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54003e00 con 0x7f5c60107670 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.380+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6006aa80 con 0x7f5c60107670 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.381+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5c6006b010 con 0x7f5c60107670 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.382+0000 7f5c47fff640 1 -- 172.21.15.111:0/3779665792 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 103145+0+0 (secure 0 0 0) 0x7f5c54018430 con 0x7f5c60107670 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.382+0000 7f5c47fff640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c4807ae00 0x7f5c4807d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.382+0000 7f5c5ed9d640 1 -- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c4807ae00 msgr2=0x7f5c4807d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3306957597 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.382+0000 7f5c47fff640 1 -- 172.21.15.111:0/3779665792 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f5c5409e700 con 0x7f5c60107670 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.382+0000 7f5c5ed9d640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c4807ae00 0x7f5c4807d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.382+0000 7f5c47fff640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] conn(0x7f5c48084850 0x7f5c48086cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.382+0000 7f5c47fff640 1 -- 172.21.15.111:0/3779665792 --> [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c48087380 con 0x7f5c48084850 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.382+0000 7f5c47fff640 1 -- 172.21.15.111:0/3779665792 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f5c540a5050 con 0x7f5c60107670 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.383+0000 7f5c5f59e640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] conn(0x7f5c48084850 0x7f5c48086cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.383+0000 7f5c5f59e640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] conn(0x7f5c48084850 0x7f5c48086cb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.383+0000 7f5c47fff640 1 -- 172.21.15.111:0/3779665792 <== osd.0 v2:172.21.15.111:6802/1090026984 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c48087380 con 0x7f5c48084850 2024-09-16T08:59:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.398+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 --> [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5c6006b410 con 0x7f5c48084850 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c47fff640 1 -- 172.21.15.111:0/3779665792 <== osd.0 v2:172.21.15.111:6802/1090026984 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5c6006b410 con 0x7f5c48084850 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] conn(0x7f5c48084850 msgr2=0x7f5c48086cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6802/1090026984,v1:172.21.15.111:6803/1090026984] conn(0x7f5c48084850 0x7f5c48086cb0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c4807ae00 msgr2=0x7f5c4807d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f5c4807ae00 0x7f5c4807d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c60107670 msgr2=0x7f5c60197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 --2- 172.21.15.111:0/3779665792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c60107670 0x7f5c60197bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5c54005fd0 tx=0x7f5c5400ee50 comp rx=0 tx=0).stop 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c5f59e640 1 -- 172.21.15.111:0/3779665792 reap_dead start 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 shutdown_connections 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 >> 172.21.15.111:0/3779665792 conn(0x7f5c600fd6d0 msgr2=0x7f5c600ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 shutdown_connections 2024-09-16T08:59:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:28.399+0000 7f5c6504d640 1 -- 172.21.15.111:0/3779665792 wait complete. 2024-09-16T08:59:29.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.352+0000 7f2046af7640 1 -- 172.21.15.111:0/1932435194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040103b00 msgr2=0x7f2040103ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:29.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.352+0000 7f2046af7640 1 --2- 172.21.15.111:0/1932435194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040103b00 0x7f2040103ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2034009900 tx=0x7f203402a410 comp rx=0 tx=0).stop 2024-09-16T08:59:29.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.352+0000 7f2046af7640 1 -- 172.21.15.111:0/1932435194 shutdown_connections 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.352+0000 7f2046af7640 1 --2- 172.21.15.111:0/1932435194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f204010d0b0 0x7f204010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.352+0000 7f2046af7640 1 --2- 172.21.15.111:0/1932435194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2040104500 0x7f204010cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.352+0000 7f2046af7640 1 --2- 172.21.15.111:0/1932435194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040103b00 0x7f2040103ee0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.352+0000 7f2046af7640 1 -- 172.21.15.111:0/1932435194 >> 172.21.15.111:0/1932435194 conn(0x7f20400fd6d0 msgr2=0x7f20400ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.352+0000 7f2046af7640 1 -- 172.21.15.111:0/1932435194 shutdown_connections 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.352+0000 7f2046af7640 1 -- 172.21.15.111:0/1932435194 wait complete. 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2046af7640 1 Processor -- start 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2046af7640 1 -- start start 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2046af7640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2040103b00 0x7f20401987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2046af7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040104500 0x7f2040198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2046af7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f204010d0b0 0x7f204019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2046af7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2040112200 con 0x7f2040104500 2024-09-16T08:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2046af7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2040112080 con 0x7f204010d0b0 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2046af7640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f2040112380 con 0x7f2040103b00 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2044893640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2040103b00 0x7f20401987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f203ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040104500 0x7f2040198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2044893640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2040103b00 0x7f20401987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:42408/0 (socket says 172.21.15.111:42408) 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2044893640 1 -- 172.21.15.111:0/1824844875 learned_addr learned my addr 172.21.15.111:0/1824844875 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f203ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040104500 0x7f2040198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35510/0 (socket says 172.21.15.111:35510) 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.353+0000 7f2045094640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f204010d0b0 0x7f204019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f203ffff640 1 -- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2040103b00 msgr2=0x7f20401987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f203ffff640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2040103b00 0x7f20401987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f203ffff640 1 -- 172.21.15.111:0/1824844875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f204010d0b0 msgr2=0x7f204019e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f203ffff640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f204010d0b0 0x7f204019e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f203ffff640 1 -- 172.21.15.111:0/1824844875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204019e860 con 0x7f2040104500 2024-09-16T08:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f2044893640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2040103b00 0x7f20401987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f2045094640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f204010d0b0 0x7f204019e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f203ffff640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040104500 0x7f2040198cf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f202c00e9f0 tx=0x7f202c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f203dffb640 1 -- 172.21.15.111:0/1824844875 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c00cd80 con 0x7f2040104500 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f203dffb640 1 -- 172.21.15.111:0/1824844875 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f202c004470 con 0x7f2040104500 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204019eb50 con 0x7f2040104500 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f203dffb640 1 -- 172.21.15.111:0/1824844875 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c003e10 con 0x7f2040104500 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.354+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20401a53e0 con 0x7f2040104500 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.355+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2040063860 con 0x7f2040104500 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.356+0000 7f203dffb640 1 -- 172.21.15.111:0/1824844875 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) ==== 103145+0+0 (secure 0 0 0) 0x7f202c004020 con 0x7f2040104500 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.356+0000 7f203dffb640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f203007ae00 0x7f203007d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.356+0000 7f2044893640 1 -- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f203007ae00 msgr2=0x7f203007d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3306957597 2024-09-16T08:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.356+0000 7f203dffb640 1 -- 172.21.15.111:0/1824844875 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7148+0+0 (secure 0 0 0) 0x7f202c09e580 con 0x7f2040104500 2024-09-16T08:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.356+0000 7f2044893640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f203007ae00 0x7f203007d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T08:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.356+0000 7f203dffb640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] conn(0x7f2030084950 0x7f2030086db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.356+0000 7f203dffb640 1 -- 172.21.15.111:0/1824844875 --> [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f202c01ee50 con 0x7f2030084950 2024-09-16T08:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.356+0000 7f203dffb640 1 -- 172.21.15.111:0/1824844875 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f202c0a5050 con 0x7f2040104500 2024-09-16T08:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.357+0000 7f2045094640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] conn(0x7f2030084950 0x7f2030086db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.357+0000 7f2045094640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] conn(0x7f2030084950 0x7f2030086db0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.358+0000 7f203dffb640 1 -- 172.21.15.111:0/1824844875 <== osd.4 v2:172.21.15.114:6800/3634478750 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f202c01ee50 con 0x7f2030084950 2024-09-16T08:59:29.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.371+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 --> [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2040063c60 con 0x7f2030084950 2024-09-16T08:59:29.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f203dffb640 1 -- 172.21.15.111:0/1824844875 <== osd.4 v2:172.21.15.114:6800/3634478750 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2040063c60 con 0x7f2030084950 2024-09-16T08:59:29.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 >> [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] conn(0x7f2030084950 msgr2=0x7f2030086db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:29.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.114:6800/3634478750,v1:172.21.15.114:6801/3634478750] conn(0x7f2030084950 0x7f2030086db0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:29.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f203007ae00 msgr2=0x7f203007d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T08:59:29.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:6800/3306957597,v1:172.21.15.111:6801/3306957597] conn(0x7f203007ae00 0x7f203007d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:29.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040104500 msgr2=0x7f2040198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 --2- 172.21.15.111:0/1824844875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040104500 0x7f2040198cf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f202c00e9f0 tx=0x7f202c00eeb0 comp rx=0 tx=0).stop 2024-09-16T08:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2045094640 1 -- 172.21.15.111:0/1824844875 reap_dead start 2024-09-16T08:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 shutdown_connections 2024-09-16T08:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 >> 172.21.15.111:0/1824844875 conn(0x7f20400fd6d0 msgr2=0x7f20400fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 shutdown_connections 2024-09-16T08:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:29.372+0000 7f2046af7640 1 -- 172.21.15.111:0/1824844875 wait complete. 2024-09-16T08:59:29.881 INFO:teuthology.orchestra.run.smithi111.stdout:34359738419 2024-09-16T08:59:29.882 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-16T08:59:30.287 INFO:teuthology.orchestra.run.smithi111.stdout:111669149729 2024-09-16T08:59:30.288 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-16T08:59:30.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[29803]: Standby manager daemon x restarted 2024-09-16T08:59:30.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[29803]: Standby manager daemon x started 2024-09-16T08:59:30.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[29803]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T08:59:30.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[29803]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T08:59:30.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[29803]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T08:59:30.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[29803]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T08:59:30.514 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[40056]: Standby manager daemon x restarted 2024-09-16T08:59:30.514 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[40056]: Standby manager daemon x started 2024-09-16T08:59:30.515 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[40056]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T08:59:30.515 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[40056]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T08:59:30.515 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[40056]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T08:59:30.515 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:30 smithi111 ceph-mon[40056]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T08:59:30.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-mon[33315]: Standby manager daemon x restarted 2024-09-16T08:59:30.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-mon[33315]: Standby manager daemon x started 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-mon[33315]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-mon[33315]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-mon[33315]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-mon[33315]: from='mgr.? 172.21.15.114:0/2799745907' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x[34799]: [16/Sep/2024:08:59:30] ENGINE Bus STARTING 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x[34799]: CherryPy Checker: 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x[34799]: The Application mounted at '' has an empty config. 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x[34799]: 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x[34799]: [16/Sep/2024:08:59:30] ENGINE Serving on http://:::9283 2024-09-16T08:59:30.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 08:59:30 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x[34799]: [16/Sep/2024:08:59:30] ENGINE Bus STARTED 2024-09-16T08:59:31.172 INFO:teuthology.orchestra.run.smithi111.stdout:158913789972 2024-09-16T08:59:31.172 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-16T08:59:31.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-16T08:59:31.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: Active manager daemon y restarted 2024-09-16T08:59:31.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: Activating manager daemon y 2024-09-16T08:59:31.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: osdmap e63: 8 total, 8 up, 8 in 2024-09-16T08:59:31.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: mgrmap e19: y(active, starting, since 0.0138437s), standbys: x 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:59:31.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: Manager daemon y is now available 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T08:59:31.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: Active manager daemon y restarted 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: Activating manager daemon y 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: osdmap e63: 8 total, 8 up, 8 in 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: mgrmap e19: y(active, starting, since 0.0138437s), standbys: x 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:59:31.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: Manager daemon y is now available 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T08:59:31.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T08:59:31.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: Active manager daemon y restarted 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: Activating manager daemon y 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: osdmap e63: 8 total, 8 up, 8 in 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: mgrmap e19: y(active, starting, since 0.0138437s), standbys: x 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-16T08:59:31.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: Manager daemon y is now available 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T08:59:31.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:31.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T08:59:31.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T08:59:31.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-16T08:59:31.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-16T08:59:31.655 INFO:teuthology.orchestra.run.smithi111.stdout:133143986203 2024-09-16T08:59:31.655 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-16T08:59:31.741 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:31.742 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:31.742 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:31.760 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:08:59:31] ENGINE Bus STARTING 2024-09-16T08:59:31.760 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: CherryPy Checker: 2024-09-16T08:59:31.760 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: The Application mounted at '' has an empty config. 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:08:59:31] ENGINE Serving on http://:::9283 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:08:59:31] ENGINE Bus STARTED 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: 2024-09-16T08:59:31.756+0000 7f678fc2a640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:08:59:31] ENGINE Error in HTTPServer.serve 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: Traceback (most recent call last): 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self._connections.run(self.expiration_interval) 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self._run(expiration_interval) 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T08:59:31.761 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T08:59:31.762 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: s = self.context.wrap_socket( 2024-09-16T08:59:31.762 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T08:59:31.762 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: return self.sslsocket_class._create( 2024-09-16T08:59:31.762 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T08:59:31.762 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self.do_handshake() 2024-09-16T08:59:31.762 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T08:59:31.762 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: self._sslobj.do_handshake() 2024-09-16T08:59:31.762 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T08:59:31.762 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:31 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: 2024-09-16T08:59:32.305 INFO:teuthology.orchestra.run.smithi111.stdout:210453397515 2024-09-16T08:59:32.305 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-16T08:59:32.325 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:32.551 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[29803]: mgrmap e20: y(active, since 1.01466s), standbys: x 2024-09-16T08:59:32.551 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[29803]: [16/Sep/2024:08:59:31] ENGINE Bus STARTING 2024-09-16T08:59:32.551 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[29803]: [16/Sep/2024:08:59:31] ENGINE Error in HTTPServer.serve 2024-09-16T08:59:32.551 INFO:journalctl@ceph.mon.a.smithi111.stdout: Traceback (most recent call last): 2024-09-16T08:59:32.551 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T08:59:32.551 INFO:journalctl@ceph.mon.a.smithi111.stdout: self._connections.run(self.expiration_interval) 2024-09-16T08:59:32.551 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T08:59:32.551 INFO:journalctl@ceph.mon.a.smithi111.stdout: self._run(expiration_interval) 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: s = self.context.wrap_socket( 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: return self.sslsocket_class._create( 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: self.do_handshake() 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: self._sslobj.do_handshake() 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T08:59:32.552 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[29803]: [16/Sep/2024:08:59:31] ENGINE Serving on https://172.21.15.111:7150 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[29803]: [16/Sep/2024:08:59:31] ENGINE Serving on http://172.21.15.111:8765 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[29803]: [16/Sep/2024:08:59:31] ENGINE Bus STARTED 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[40056]: mgrmap e20: y(active, since 1.01466s), standbys: x 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[40056]: [16/Sep/2024:08:59:31] ENGINE Bus STARTING 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[40056]: [16/Sep/2024:08:59:31] ENGINE Error in HTTPServer.serve 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.c.smithi111.stdout: Traceback (most recent call last): 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.c.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.c.smithi111.stdout: self._connections.run(self.expiration_interval) 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.c.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.c.smithi111.stdout: self._run(expiration_interval) 2024-09-16T08:59:32.553 INFO:journalctl@ceph.mon.c.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: s = self.context.wrap_socket( 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: return self.sslsocket_class._create( 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: self.do_handshake() 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: self._sslobj.do_handshake() 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[40056]: [16/Sep/2024:08:59:31] ENGINE Serving on https://172.21.15.111:7150 2024-09-16T08:59:32.554 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[40056]: [16/Sep/2024:08:59:31] ENGINE Serving on http://172.21.15.111:8765 2024-09-16T08:59:32.555 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:32 smithi111 ceph-mon[40056]: [16/Sep/2024:08:59:31] ENGINE Bus STARTED 2024-09-16T08:59:32.792 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:32 smithi114 ceph-mon[33315]: mgrmap e20: y(active, since 1.01466s), standbys: x 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:32 smithi114 ceph-mon[33315]: [16/Sep/2024:08:59:31] ENGINE Bus STARTING 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:32 smithi114 ceph-mon[33315]: [16/Sep/2024:08:59:31] ENGINE Error in HTTPServer.serve 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: Traceback (most recent call last): 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: self._run(expiration_interval) 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T08:59:32.793 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout: s = self.context.wrap_socket( 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout: return self.sslsocket_class._create( 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout: self.do_handshake() 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout: self._sslobj.do_handshake() 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:32 smithi114 ceph-mon[33315]: [16/Sep/2024:08:59:31] ENGINE Serving on https://172.21.15.111:7150 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:32 smithi114 ceph-mon[33315]: [16/Sep/2024:08:59:31] ENGINE Serving on http://172.21.15.111:8765 2024-09-16T08:59:32.794 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:32 smithi114 ceph-mon[33315]: [16/Sep/2024:08:59:31] ENGINE Bus STARTED 2024-09-16T08:59:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.100+0000 7f02b4f5f640 1 -- 172.21.15.111:0/3940034290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b0107670 msgr2=0x7f02b0107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.100+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/3940034290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b0107670 0x7f02b0107ab0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009900 tx=0x7f02a402a410 comp rx=0 tx=0).stop 2024-09-16T08:59:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.100+0000 7f02b4f5f640 1 -- 172.21.15.111:0/3940034290 shutdown_connections 2024-09-16T08:59:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.100+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/3940034290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0107ff0 0x7f02b010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.100+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/3940034290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b0107670 0x7f02b0107ab0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.100+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/3940034290 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b0101c60 0x7f02b0102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.100+0000 7f02b4f5f640 1 -- 172.21.15.111:0/3940034290 >> 172.21.15.111:0/3940034290 conn(0x7f02b00fd6d0 msgr2=0x7f02b00ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.100+0000 7f02b4f5f640 1 -- 172.21.15.111:0/3940034290 shutdown_connections 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.100+0000 7f02b4f5f640 1 -- 172.21.15.111:0/3940034290 wait complete. 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02b4f5f640 1 Processor -- start 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02b4f5f640 1 -- start start 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02b4f5f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0101c60 0x7f02b010e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02b4f5f640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b0107670 0x7f02b010a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02b4f5f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b0107ff0 0x7f02b010acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02b4f5f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f02b0111110 con 0x7f02b0107ff0 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02b4f5f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f02b0110f90 con 0x7f02b0101c60 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02b4f5f640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f02b0111290 con 0x7f02b0107670 2024-09-16T08:59:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02ae59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0101c60 0x7f02b010e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02aed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b0107ff0 0x7f02b010acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02add9b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b0107670 0x7f02b010a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02ae59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0101c60 0x7f02b010e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.111:35774/0 (socket says 172.21.15.111:35774) 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02ae59c640 1 -- 172.21.15.111:0/4054691830 learned_addr learned my addr 172.21.15.111:0/4054691830 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.101+0000 7f02add9b640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b0107670 0x7f02b010a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:42524/0 (socket says 172.21.15.111:42524) 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02add9b640 1 -- 172.21.15.111:0/4054691830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0101c60 msgr2=0x7f02b010e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02add9b640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0101c60 0x7f02b010e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02add9b640 1 -- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b0107ff0 msgr2=0x7f02b010acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02add9b640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b0107ff0 0x7f02b010acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02add9b640 1 -- 172.21.15.111:0/4054691830 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b010b550 con 0x7f02b0107670 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02aed9d640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b0107ff0 0x7f02b010acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02ae59c640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0101c60 0x7f02b010e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02add9b640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b0107670 0x7f02b010a780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02a402a930 tx=0x7f02a4032af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f029f7fe640 1 -- 172.21.15.111:0/4054691830 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a403f030 con 0x7f02b0107670 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f029f7fe640 1 -- 172.21.15.111:0/4054691830 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f02a4003a40 con 0x7f02b0107670 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b01ab250 con 0x7f02b0107670 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f029f7fe640 1 -- 172.21.15.111:0/4054691830 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4035070 con 0x7f02b0107670 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.102+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b01ab7e0 con 0x7f02b0107670 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.103+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b0102040 con 0x7f02b0107670 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.104+0000 7f029f7fe640 1 -- 172.21.15.111:0/4054691830 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 21) ==== 103082+0+0 (secure 0 0 0) 0x7f02a4042030 con 0x7f02b0107670 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.105+0000 7f029f7fe640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f029807ad80 0x7f029807d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.105+0000 7f02ae59c640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f029807ad80 0x7f029807d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.105+0000 7f029f7fe640 1 -- 172.21.15.111:0/4054691830 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7f02a40becc0 con 0x7f02b0107670 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.105+0000 7f02ae59c640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f029807ad80 0x7f029807d240 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f02a00095f0 tx=0x7f02a0005f90 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.109+0000 7f029f7fe640 1 -- 172.21.15.111:0/4054691830 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f02a408a0b0 con 0x7f02b0107670 2024-09-16T08:59:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/2197804198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4101c80 msgr2=0x7f1ec4102060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/2197804198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4101c80 0x7f1ec4102060 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009860 tx=0x7f1eb402a340 comp rx=0 tx=0).stop 2024-09-16T08:59:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/2197804198 shutdown_connections 2024-09-16T08:59:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/2197804198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec4108010 0x7f1ec410e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/2197804198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1ec4107690 0x7f1ec4107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/2197804198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4101c80 0x7f1ec4102060 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/2197804198 >> 172.21.15.111:0/2197804198 conn(0x7f1ec40fd6f0 msgr2=0x7f1ec40ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/2197804198 shutdown_connections 2024-09-16T08:59:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/2197804198 wait complete. 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 Processor -- start 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.000+0000 7f1ecb0ac640 1 -- start start 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ecb0ac640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec4101c80 0x7f1ec41987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ecb0ac640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1ec4107690 0x7f1ec4198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ecb0ac640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4108010 0x7f1ec419e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ecb0ac640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec41122a0 con 0x7f1ec4108010 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ecb0ac640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4112120 con 0x7f1ec4101c80 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ecb0ac640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec4112420 con 0x7f1ec4107690 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec3fff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1ec4107690 0x7f1ec4198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec9649640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4108010 0x7f1ec419e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec3fff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1ec4107690 0x7f1ec4198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:42500/0 (socket says 172.21.15.111:42500) 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec9649640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4108010 0x7f1ec419e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35620/0 (socket says 172.21.15.111:35620) 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec3fff640 1 -- 172.21.15.111:0/1899600976 learned_addr learned my addr 172.21.15.111:0/1899600976 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec8e48640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec4101c80 0x7f1ec41987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec3fff640 1 -- 172.21.15.111:0/1899600976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec4101c80 msgr2=0x7f1ec41987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec3fff640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec4101c80 0x7f1ec41987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec3fff640 1 -- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4108010 msgr2=0x7f1ec419e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec3fff640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4108010 0x7f1ec419e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec3fff640 1 -- 172.21.15.111:0/1899600976 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec419e860 con 0x7f1ec4107690 2024-09-16T08:59:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec9649640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4108010 0x7f1ec419e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec3fff640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1ec4107690 0x7f1ec4198cf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800e9c0 tx=0x7f1eb800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec1ffb640 1 -- 172.21.15.111:0/1899600976 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800cde0 con 0x7f1ec4107690 2024-09-16T08:59:33.254 INFO:teuthology.orchestra.run.smithi111.stdout:184683593745 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.001+0000 7f1ec1ffb640 1 -- 172.21.15.111:0/1899600976 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f1eb8004470 con 0x7f1ec4107690 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.002+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec419eb50 con 0x7f1ec4107690 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.002+0000 7f1ec1ffb640 1 -- 172.21.15.111:0/1899600976 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8003e50 con 0x7f1ec4107690 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.002+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec41a53e0 con 0x7f1ec4107690 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.003+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec4102060 con 0x7f1ec4107690 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.003+0000 7f1ec1ffb640 1 -- 172.21.15.111:0/1899600976 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 21) ==== 103082+0+0 (secure 0 0 0) 0x7f1eb8004000 con 0x7f1ec4107690 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.004+0000 7f1ec1ffb640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f1eac07af20 0x7f1eac07d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.004+0000 7f1ec1ffb640 1 -- 172.21.15.111:0/1899600976 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7f1eb809e580 con 0x7f1ec4107690 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.004+0000 7f1ec8e48640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f1eac07af20 0x7f1eac07d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.004+0000 7f1ec8e48640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f1eac07af20 0x7f1eac07d3e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1eb402a930 tx=0x7f1eb40058d0 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.008+0000 7f1ec1ffb640 1 -- 172.21.15.111:0/1899600976 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f1eb8069990 con 0x7f1ec4107690 2024-09-16T08:59:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.199+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f1ec4109df0 con 0x7f1ec4107690 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.199+0000 7f1ec1ffb640 1 -- 172.21.15.111:0/1899600976 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1eb806e840 con 0x7f1ec4107690 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f1eac07af20 msgr2=0x7f1eac07d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f1eac07af20 0x7f1eac07d3e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1eb402a930 tx=0x7f1eb40058d0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1ec4107690 msgr2=0x7f1ec4198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1ec4107690 0x7f1ec4198cf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800e9c0 tx=0x7f1eb800ee80 comp rx=0 tx=0).stop 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 shutdown_connections 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f1eac07af20 0x7f1eac07d3e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ec4108010 0x7f1ec419e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1ec4107690 0x7f1ec4198cf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 --2- 172.21.15.111:0/1899600976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec4101c80 0x7f1ec41987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 >> 172.21.15.111:0/1899600976 conn(0x7f1ec40fd6f0 msgr2=0x7f1ec40ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 shutdown_connections 2024-09-16T08:59:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.203+0000 7f1ecb0ac640 1 -- 172.21.15.111:0/1899600976 wait complete. 2024-09-16T08:59:33.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.316+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f02b01aba40 con 0x7f02b0107670 2024-09-16T08:59:33.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.317+0000 7f029f7fe640 1 -- 172.21.15.111:0/4054691830 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f02a408ef60 con 0x7f02b0107670 2024-09-16T08:59:33.318 INFO:teuthology.orchestra.run.smithi111.stdout:77309411367 2024-09-16T08:59:33.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f029807ad80 msgr2=0x7f029807d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f029807ad80 0x7f029807d240 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f02a00095f0 tx=0x7f02a0005f90 comp rx=0 tx=0).stop 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b0107670 msgr2=0x7f02b010a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b0107670 0x7f02b010a780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02a402a930 tx=0x7f02a4032af0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 shutdown_connections 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f029807ad80 0x7f029807d240 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b0107ff0 0x7f02b010acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b0107670 0x7f02b010a780 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 --2- 172.21.15.111:0/4054691830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b0101c60 0x7f02b010e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 >> 172.21.15.111:0/4054691830 conn(0x7f02b00fd6d0 msgr2=0x7f02b006b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 shutdown_connections 2024-09-16T08:59:33.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:33.319+0000 7f02b4f5f640 1 -- 172.21.15.111:0/4054691830 wait complete. 2024-09-16T08:59:33.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:08:59:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T08:59:33.695 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:33.732 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:33.816 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:33 smithi111 ceph-mon[40056]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:59:33.816 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:33 smithi111 ceph-mon[40056]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-16T08:59:33.816 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1899600976' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-16T08:59:33.817 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4054691830' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-16T08:59:33.817 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:33 smithi111 ceph-mon[29803]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:59:33.817 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:33 smithi111 ceph-mon[29803]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-16T08:59:33.817 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1899600976' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-16T08:59:33.817 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4054691830' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-16T08:59:33.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:33 smithi114 ceph-mon[33315]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:59:33.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:33 smithi114 ceph-mon[33315]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-16T08:59:33.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1899600976' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-16T08:59:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4054691830' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-16T08:59:34.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.282+0000 7f42c11ee640 1 -- 172.21.15.111:0/3893896749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc101c80 msgr2=0x7f42bc102060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.282+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3893896749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc101c80 0x7f42bc102060 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009900 tx=0x7f42ac02a410 comp rx=0 tx=0).stop 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.282+0000 7f42c11ee640 1 -- 172.21.15.111:0/3893896749 shutdown_connections 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.282+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3893896749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc108010 0x7f42bc10e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.282+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3893896749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc107690 0x7f42bc107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.282+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3893896749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc101c80 0x7f42bc102060 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.282+0000 7f42c11ee640 1 -- 172.21.15.111:0/3893896749 >> 172.21.15.111:0/3893896749 conn(0x7f42bc0fd6f0 msgr2=0x7f42bc0ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.282+0000 7f42c11ee640 1 -- 172.21.15.111:0/3893896749 shutdown_connections 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.282+0000 7f42c11ee640 1 -- 172.21.15.111:0/3893896749 wait complete. 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42c11ee640 1 Processor -- start 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42c11ee640 1 -- start start 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42c11ee640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc101c80 0x7f42bc197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42c11ee640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc107690 0x7f42bc197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42c11ee640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc108010 0x7f42bc19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:34.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42c11ee640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc1111a0 con 0x7f42bc108010 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42c11ee640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc111020 con 0x7f42bc101c80 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42c11ee640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f42bc111320 con 0x7f42bc107690 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bb59e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc108010 0x7f42bc19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42b259c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc107690 0x7f42bc197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc101c80 0x7f42bc197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42b259c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc107690 0x7f42bc197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:42546/0 (socket says 172.21.15.111:42546) 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc101c80 0x7f42bc197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.111:35796/0 (socket says 172.21.15.111:35796) 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42b259c640 1 -- 172.21.15.111:0/3158559756 learned_addr learned my addr 172.21.15.111:0/3158559756 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bb59e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc108010 0x7f42bc19cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35652/0 (socket says 172.21.15.111:35652) 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bb59e640 1 -- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc107690 msgr2=0x7f42bc197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bb59e640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc107690 0x7f42bc197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bb59e640 1 -- 172.21.15.111:0/3158559756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc101c80 msgr2=0x7f42bc197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bb59e640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc101c80 0x7f42bc197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bb59e640 1 -- 172.21.15.111:0/3158559756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc19d7c0 con 0x7f42bc108010 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42b259c640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc107690 0x7f42bc197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.283+0000 7f42bad9d640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc101c80 0x7f42bc197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.284+0000 7f42bb59e640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc108010 0x7f42bc19cff0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f42b400ede0 tx=0x7f42b400b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.284+0000 7f42b8d99640 1 -- 172.21.15.111:0/3158559756 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4003970 con 0x7f42bc108010 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.284+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42bc19dab0 con 0x7f42bc108010 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.284+0000 7f42b8d99640 1 -- 172.21.15.111:0/3158559756 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f42b4004020 con 0x7f42bc108010 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.284+0000 7f42b8d99640 1 -- 172.21.15.111:0/3158559756 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4017650 con 0x7f42bc108010 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.284+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42bc06b460 con 0x7f42bc108010 2024-09-16T08:59:34.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.285+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42bc102060 con 0x7f42bc108010 2024-09-16T08:59:34.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.286+0000 7f42b8d99640 1 -- 172.21.15.111:0/3158559756 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 21) ==== 103082+0+0 (secure 0 0 0) 0x7f42b4003b10 con 0x7f42bc108010 2024-09-16T08:59:34.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.286+0000 7f42b8d99640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f42a807ad80 0x7f42a807d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:34.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.286+0000 7f42b8d99640 1 -- 172.21.15.111:0/3158559756 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7f42b409fd80 con 0x7f42bc108010 2024-09-16T08:59:34.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.286+0000 7f42bad9d640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f42a807ad80 0x7f42a807d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:34.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.287+0000 7f42bad9d640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f42a807ad80 0x7f42a807d240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f42ac000c00 tx=0x7f42ac0317b0 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:34.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.290+0000 7f42b8d99640 1 -- 172.21.15.111:0/3158559756 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f42b406b110 con 0x7f42bc108010 2024-09-16T08:59:34.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.479+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f42bc06bf40 con 0x7f42bc108010 2024-09-16T08:59:34.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.479+0000 7f42b8d99640 1 -- 172.21.15.111:0/3158559756 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f42b406ffc0 con 0x7f42bc108010 2024-09-16T08:59:34.480 INFO:teuthology.orchestra.run.smithi111.stdout:111669149730 2024-09-16T08:59:34.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f42a807ad80 msgr2=0x7f42a807d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f42a807ad80 0x7f42a807d240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f42ac000c00 tx=0x7f42ac0317b0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc108010 msgr2=0x7f42bc19cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc108010 0x7f42bc19cff0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f42b400ede0 tx=0x7f42b400b690 comp rx=0 tx=0).stop 2024-09-16T08:59:34.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 shutdown_connections 2024-09-16T08:59:34.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f42a807ad80 0x7f42a807d240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc108010 0x7f42bc19cff0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f42bc107690 0x7f42bc197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 --2- 172.21.15.111:0/3158559756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42bc101c80 0x7f42bc197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.481+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 >> 172.21.15.111:0/3158559756 conn(0x7f42bc0fd6f0 msgr2=0x7f42bc0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:34.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.482+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 shutdown_connections 2024-09-16T08:59:34.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.482+0000 7f42c11ee640 1 -- 172.21.15.111:0/3158559756 wait complete. 2024-09-16T08:59:34.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411367 for osd.2 2024-09-16T08:59:34.849 DEBUG:teuthology.parallel:result is None 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.487+0000 7fb794815640 1 -- 172.21.15.111:0/2419965315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 msgr2=0x7fb790107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.487+0000 7fb794815640 1 --2- 172.21.15.111:0/2419965315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 0x7fb790107ab0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb784009860 tx=0x7fb78402a340 comp rx=0 tx=0).stop 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.487+0000 7fb794815640 1 -- 172.21.15.111:0/2419965315 shutdown_connections 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.487+0000 7fb794815640 1 --2- 172.21.15.111:0/2419965315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb790107ff0 0x7fb79010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.487+0000 7fb794815640 1 --2- 172.21.15.111:0/2419965315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 0x7fb790107ab0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.487+0000 7fb794815640 1 --2- 172.21.15.111:0/2419965315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb790101c60 0x7fb790102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.487+0000 7fb794815640 1 -- 172.21.15.111:0/2419965315 >> 172.21.15.111:0/2419965315 conn(0x7fb7900fd710 msgr2=0x7fb7900ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.487+0000 7fb794815640 1 -- 172.21.15.111:0/2419965315 shutdown_connections 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.487+0000 7fb794815640 1 -- 172.21.15.111:0/2419965315 wait complete. 2024-09-16T08:59:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.488+0000 7fb794815640 1 Processor -- start 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.488+0000 7fb794815640 1 -- start start 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.488+0000 7fb794815640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb790101c60 0x7fb7901976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.488+0000 7fb794815640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 0x7fb790197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.488+0000 7fb794815640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb790107ff0 0x7fb79019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.488+0000 7fb794815640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb790111110 con 0x7fb790107ff0 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.488+0000 7fb794815640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb790110f90 con 0x7fb790101c60 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.488+0000 7fb794815640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fb790111290 con 0x7fb790107670 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78dd9b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 0x7fb790197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78dd9b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 0x7fb790197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:42562/0 (socket says 172.21.15.111:42562) 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78dd9b640 1 -- 172.21.15.111:0/4215307615 learned_addr learned my addr 172.21.15.111:0/4215307615 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78ed9d640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb790107ff0 0x7fb79019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78e59c640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb790101c60 0x7fb7901976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78dd9b640 1 -- 172.21.15.111:0/4215307615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb790101c60 msgr2=0x7fb7901976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78dd9b640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb790101c60 0x7fb7901976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78dd9b640 1 -- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb790107ff0 msgr2=0x7fb79019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78dd9b640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb790107ff0 0x7fb79019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78dd9b640 1 -- 172.21.15.111:0/4215307615 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb79019d750 con 0x7fb790107670 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78e59c640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb790101c60 0x7fb7901976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78ed9d640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb790107ff0 0x7fb79019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb78dd9b640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 0x7fb790197be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb784035040 tx=0x7fb7840305f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb77f7fe640 1 -- 172.21.15.111:0/4215307615 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb784032e50 con 0x7fb790107670 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb79019d9e0 con 0x7fb790107670 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb77f7fe640 1 -- 172.21.15.111:0/4215307615 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fb78403d030 con 0x7fb790107670 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb77f7fe640 1 -- 172.21.15.111:0/4215307615 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb784030d10 con 0x7fb790107670 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.489+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb79006b2a0 con 0x7fb790107670 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.490+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb790102040 con 0x7fb790107670 2024-09-16T08:59:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.491+0000 7fb77f7fe640 1 -- 172.21.15.111:0/4215307615 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 21) ==== 103082+0+0 (secure 0 0 0) 0x7fb784004050 con 0x7fb790107670 2024-09-16T08:59:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.492+0000 7fb77f7fe640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb77807af20 0x7fb77807d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.492+0000 7fb78e59c640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb77807af20 0x7fb77807d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.492+0000 7fb77f7fe640 1 -- 172.21.15.111:0/4215307615 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7fb7840bf7f0 con 0x7fb790107670 2024-09-16T08:59:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.492+0000 7fb78e59c640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb77807af20 0x7fb77807d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb780005fd0 tx=0x7fb780009260 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.496+0000 7fb77f7fe640 1 -- 172.21.15.111:0/4215307615 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fb78408ac90 con 0x7fb790107670 2024-09-16T08:59:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.680+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fb79006bd80 con 0x7fb790107670 2024-09-16T08:59:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.680+0000 7fb77f7fe640 1 -- 172.21.15.111:0/4215307615 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb78408be20 con 0x7fb790107670 2024-09-16T08:59:34.943 INFO:teuthology.orchestra.run.smithi111.stdout:210453397516 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb77807af20 msgr2=0x7fb77807d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb77807af20 0x7fb77807d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb780005fd0 tx=0x7fb780009260 comp rx=0 tx=0).stop 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 msgr2=0x7fb790197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 0x7fb790197be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb784035040 tx=0x7fb7840305f0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 shutdown_connections 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb77807af20 0x7fb77807d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb790107ff0 0x7fb79019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb790107670 0x7fb790197be0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 --2- 172.21.15.111:0/4215307615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb790101c60 0x7fb7901976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 >> 172.21.15.111:0/4215307615 conn(0x7fb7900fd710 msgr2=0x7fb7900ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 shutdown_connections 2024-09-16T08:59:34.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:34.682+0000 7fb794815640 1 -- 172.21.15.111:0/4215307615 wait complete. 2024-09-16T08:59:34.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:34.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:34.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:34.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:34.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3158559756' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4215307615' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:34.946 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3158559756' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-16T08:59:34.947 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4215307615' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-16T08:59:35.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3158559756' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-16T08:59:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4215307615' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-16T08:59:35.470 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397515 got 210453397516 for osd.7 2024-09-16T08:59:35.470 DEBUG:teuthology.parallel:result is None 2024-09-16T08:59:36.065 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:35 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi114 to 3760M 2024-09-16T08:59:36.065 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:35 smithi111 ceph-mon[29803]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:59:36.065 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:35 smithi111 ceph-mon[29803]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-16T08:59:36.066 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:35 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi114 to 3760M 2024-09-16T08:59:36.066 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:35 smithi111 ceph-mon[40056]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:59:36.066 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:35 smithi111 ceph-mon[40056]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-16T08:59:36.072 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:36.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:35 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi114 to 3760M 2024-09-16T08:59:36.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:35 smithi114 ceph-mon[33315]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:59:36.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:35 smithi114 ceph-mon[33315]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-16T08:59:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 -- 172.21.15.111:0/3743678218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 msgr2=0x7f781c102080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 --2- 172.21.15.111:0/3743678218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 0x7f781c102080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f780c009900 tx=0x7f780c02a410 comp rx=0 tx=0).stop 2024-09-16T08:59:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 -- 172.21.15.111:0/3743678218 shutdown_connections 2024-09-16T08:59:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 --2- 172.21.15.111:0/3743678218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f781c108030 0x7f781c10e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 --2- 172.21.15.111:0/3743678218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f781c1076b0 0x7f781c107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 --2- 172.21.15.111:0/3743678218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 0x7f781c102080 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 -- 172.21.15.111:0/3743678218 >> 172.21.15.111:0/3743678218 conn(0x7f781c0fd710 msgr2=0x7f781c0ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 -- 172.21.15.111:0/3743678218 shutdown_connections 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 -- 172.21.15.111:0/3743678218 wait complete. 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 Processor -- start 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.150+0000 7f7822394640 1 -- start start 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f7822394640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 0x7f781c197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f7822394640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f781c1076b0 0x7f781c197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f7822394640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f781c108030 0x7f781c19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f7822394640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f781c1111c0 con 0x7f781c1076b0 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f7822394640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f781c111040 con 0x7f781c108030 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f7822394640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f781c111340 con 0x7f781c101ca0 2024-09-16T08:59:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781bfff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 0x7f781c197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781bfff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 0x7f781c197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:42588/0 (socket says 172.21.15.111:42588) 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781bfff640 1 -- 172.21.15.111:0/2231566180 learned_addr learned my addr 172.21.15.111:0/2231566180 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781b7fe640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f781c1076b0 0x7f781c197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f7820931640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f781c108030 0x7f781c19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781bfff640 1 -- 172.21.15.111:0/2231566180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f781c108030 msgr2=0x7f781c19d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781bfff640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f781c108030 0x7f781c19d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781bfff640 1 -- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f781c1076b0 msgr2=0x7f781c197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781bfff640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f781c1076b0 0x7f781c197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781bfff640 1 -- 172.21.15.111:0/2231566180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781c19d7e0 con 0x7f781c101ca0 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781b7fe640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f781c1076b0 0x7f781c197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.151+0000 7f781bfff640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 0x7f781c197730 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f780c005940 tx=0x7f780c032b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.152+0000 7f78197fa640 1 -- 172.21.15.111:0/2231566180 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c03f030 con 0x7f781c101ca0 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.152+0000 7f78197fa640 1 -- 172.21.15.111:0/2231566180 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f780c003a40 con 0x7f781c101ca0 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.152+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f781c19da70 con 0x7f781c101ca0 2024-09-16T08:59:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.152+0000 7f78197fa640 1 -- 172.21.15.111:0/2231566180 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c035070 con 0x7f781c101ca0 2024-09-16T08:59:36.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.152+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f781c1a4190 con 0x7f781c101ca0 2024-09-16T08:59:36.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.153+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f781c102080 con 0x7f781c101ca0 2024-09-16T08:59:36.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.153+0000 7f78197fa640 1 -- 172.21.15.111:0/2231566180 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f780c042030 con 0x7f781c101ca0 2024-09-16T08:59:36.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.154+0000 7f78197fa640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f780407ae10 0x7f780407d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:36.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.154+0000 7f781b7fe640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f780407ae10 0x7f780407d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:36.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.154+0000 7f78197fa640 1 -- 172.21.15.111:0/2231566180 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7f780c0be7f0 con 0x7f781c101ca0 2024-09-16T08:59:36.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.154+0000 7f781b7fe640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f780407ae10 0x7f780407d2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f781c107200 tx=0x7f7810009110 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:36.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.158+0000 7f78197fa640 1 -- 172.21.15.111:0/2231566180 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f780c089b20 con 0x7f781c101ca0 2024-09-16T08:59:36.346 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.345+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f781c109e10 con 0x7f781c101ca0 2024-09-16T08:59:36.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.346+0000 7f78197fa640 1 -- 172.21.15.111:0/2231566180 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f780c08e9d0 con 0x7f781c101ca0 2024-09-16T08:59:36.347 INFO:teuthology.orchestra.run.smithi111.stdout:34359738420 2024-09-16T08:59:36.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.348+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f780407ae10 msgr2=0x7f780407d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.348+0000 7f7822394640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f780407ae10 0x7f780407d2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f781c107200 tx=0x7f7810009110 comp rx=0 tx=0).stop 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.348+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 msgr2=0x7f781c197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.348+0000 7f7822394640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 0x7f781c197730 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f780c005940 tx=0x7f780c032b70 comp rx=0 tx=0).stop 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.349+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 shutdown_connections 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.349+0000 7f7822394640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f780407ae10 0x7f780407d2d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.349+0000 7f7822394640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f781c108030 0x7f781c19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.349+0000 7f7822394640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f781c1076b0 0x7f781c197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.349+0000 7f7822394640 1 --2- 172.21.15.111:0/2231566180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f781c101ca0 0x7f781c197730 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.349+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 >> 172.21.15.111:0/2231566180 conn(0x7f781c0fd710 msgr2=0x7f781c0ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.349+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 shutdown_connections 2024-09-16T08:59:36.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.349+0000 7f7822394640 1 -- 172.21.15.111:0/2231566180 wait complete. 2024-09-16T08:59:36.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 -- 172.21.15.111:0/2618038628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1c102210 msgr2=0x7fed1c10c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:36.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 --2- 172.21.15.111:0/2618038628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1c102210 0x7fed1c10c980 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fed10009860 tx=0x7fed1002a340 comp rx=0 tx=0).stop 2024-09-16T08:59:36.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 -- 172.21.15.111:0/2618038628 shutdown_connections 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 --2- 172.21.15.111:0/2618038628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed1c10d0c0 0x7fed1c10f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 --2- 172.21.15.111:0/2618038628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1c102210 0x7fed1c10c980 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 --2- 172.21.15.111:0/2618038628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fed1c101860 0x7fed1c101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 -- 172.21.15.111:0/2618038628 >> 172.21.15.111:0/2618038628 conn(0x7fed1c076ea0 msgr2=0x7fed1c0772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 -- 172.21.15.111:0/2618038628 shutdown_connections 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 -- 172.21.15.111:0/2618038628 wait complete. 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 Processor -- start 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.843+0000 7fed214d9640 1 -- start start 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed214d9640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fed1c101860 0x7fed1c198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed214d9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed1c102210 0x7fed1c198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed214d9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1c10d0c0 0x7fed1c19e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed214d9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c112240 con 0x7fed1c10d0c0 2024-09-16T08:59:36.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed214d9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1120c0 con 0x7fed1c102210 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed214d9640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c1123c0 con 0x7fed1c101860 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1affd640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fed1c101860 0x7fed1c198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1affd640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fed1c101860 0x7fed1c198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:42608/0 (socket says 172.21.15.111:42608) 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1affd640 1 -- 172.21.15.111:0/3687925639 learned_addr learned my addr 172.21.15.111:0/3687925639 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1b7fe640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1c10d0c0 0x7fed1c19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1a7fc640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed1c102210 0x7fed1c198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1affd640 1 -- 172.21.15.111:0/3687925639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed1c102210 msgr2=0x7fed1c198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1affd640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed1c102210 0x7fed1c198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1affd640 1 -- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1c10d0c0 msgr2=0x7fed1c19e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1affd640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1c10d0c0 0x7fed1c19e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1affd640 1 -- 172.21.15.111:0/3687925639 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c19e8e0 con 0x7fed1c101860 2024-09-16T08:59:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed1affd640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fed1c101860 0x7fed1c198830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00e9c0 tx=0x7fed0c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed03fff640 1 -- 172.21.15.111:0/3687925639 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00cde0 con 0x7fed1c101860 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed03fff640 1 -- 172.21.15.111:0/3687925639 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fed0c004470 con 0x7fed1c101860 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1c19ebd0 con 0x7fed1c101860 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed03fff640 1 -- 172.21.15.111:0/3687925639 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c003e50 con 0x7fed1c101860 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.844+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1c1a5290 con 0x7fed1c101860 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.845+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed1c101c40 con 0x7fed1c101860 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.846+0000 7fed03fff640 1 -- 172.21.15.111:0/3687925639 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7fed0c004000 con 0x7fed1c101860 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.846+0000 7fed03fff640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fed0407afb0 0x7fed0407d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.847+0000 7fed1a7fc640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fed0407afb0 0x7fed0407d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.847+0000 7fed03fff640 1 -- 172.21.15.111:0/3687925639 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7fed0c09e740 con 0x7fed1c101860 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.847+0000 7fed1a7fc640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fed0407afb0 0x7fed0407d470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fed10009830 tx=0x7fed100058d0 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:36.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:36.851+0000 7fed03fff640 1 -- 172.21.15.111:0/3687925639 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fed0c0699f0 con 0x7fed1c101860 2024-09-16T08:59:36.988 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2231566180' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-16T08:59:36.989 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2231566180' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-16T08:59:37.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.036+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fed1c108040 con 0x7fed1c101860 2024-09-16T08:59:37.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.036+0000 7fed03fff640 1 -- 172.21.15.111:0/3687925639 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fed0c06e8a0 con 0x7fed1c101860 2024-09-16T08:59:37.037 INFO:teuthology.orchestra.run.smithi111.stdout:133143986204 2024-09-16T08:59:37.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fed0407afb0 msgr2=0x7fed0407d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:37.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fed0407afb0 0x7fed0407d470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fed10009830 tx=0x7fed100058d0 comp rx=0 tx=0).stop 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fed1c101860 msgr2=0x7fed1c198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fed1c101860 0x7fed1c198830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00e9c0 tx=0x7fed0c00ee80 comp rx=0 tx=0).stop 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 shutdown_connections 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fed0407afb0 0x7fed0407d470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1c10d0c0 0x7fed1c19e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed1c102210 0x7fed1c198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 --2- 172.21.15.111:0/3687925639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fed1c101860 0x7fed1c198830 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 >> 172.21.15.111:0/3687925639 conn(0x7fed1c076ea0 msgr2=0x7fed1c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 shutdown_connections 2024-09-16T08:59:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:37.039+0000 7fed214d9640 1 -- 172.21.15.111:0/3687925639 wait complete. 2024-09-16T08:59:37.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2231566180' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-16T08:59:37.618 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738419 got 34359738420 for osd.0 2024-09-16T08:59:37.618 DEBUG:teuthology.parallel:result is None 2024-09-16T08:59:37.925 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:37 smithi111 ceph-mon[29803]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:59:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3687925639' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-16T08:59:37.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:37 smithi111 ceph-mon[40056]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:59:37.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3687925639' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-16T08:59:38.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:37 smithi114 ceph-mon[33315]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-16T08:59:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3687925639' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-16T08:59:38.392 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:38.809 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986203 got 133143986204 for osd.4 2024-09-16T08:59:38.809 DEBUG:teuthology.parallel:result is None 2024-09-16T08:59:39.052 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149729 got 111669149730 for osd.3 2024-09-16T08:59:39.052 DEBUG:teuthology.parallel:result is None 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.222+0000 7f4370373640 1 -- 172.21.15.111:0/1189125700 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4368104590 msgr2=0x7f436810ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.222+0000 7f4370373640 1 --2- 172.21.15.111:0/1189125700 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4368104590 0x7f436810ca80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4358009900 tx=0x7f435802a410 comp rx=0 tx=0).stop 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.223+0000 7f4370373640 1 -- 172.21.15.111:0/1189125700 shutdown_connections 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.223+0000 7f4370373640 1 --2- 172.21.15.111:0/1189125700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f436810d0d0 0x7f436810f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.223+0000 7f4370373640 1 --2- 172.21.15.111:0/1189125700 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4368104590 0x7f436810ca80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.223+0000 7f4370373640 1 --2- 172.21.15.111:0/1189125700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4368103c70 0x7f4368104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.223+0000 7f4370373640 1 -- 172.21.15.111:0/1189125700 >> 172.21.15.111:0/1189125700 conn(0x7f43680fd6d0 msgr2=0x7f43680ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.223+0000 7f4370373640 1 -- 172.21.15.111:0/1189125700 shutdown_connections 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.223+0000 7f4370373640 1 -- 172.21.15.111:0/1189125700 wait complete. 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.223+0000 7f4370373640 1 Processor -- start 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.223+0000 7f4370373640 1 -- start start 2024-09-16T08:59:39.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f4370373640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4368103c70 0x7f43681987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f4370373640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4368104590 0x7f4368198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f4370373640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436810d0d0 0x7f436819e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f4370373640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4368112210 con 0x7f436810d0d0 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f4370373640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4368112090 con 0x7f4368103c70 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f4370373640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f4368112390 con 0x7f4368104590 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e910640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436810d0d0 0x7f436819e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436d90e640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4368104590 0x7f4368198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e910640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436810d0d0 0x7f436819e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35722/0 (socket says 172.21.15.111:35722) 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e910640 1 -- 172.21.15.111:0/849203626 learned_addr learned my addr 172.21.15.111:0/849203626 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e10f640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4368103c70 0x7f43681987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e910640 1 -- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4368104590 msgr2=0x7f4368198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e910640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4368104590 0x7f4368198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e910640 1 -- 172.21.15.111:0/849203626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4368103c70 msgr2=0x7f43681987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e910640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4368103c70 0x7f43681987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e910640 1 -- 172.21.15.111:0/849203626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436819e870 con 0x7f436810d0d0 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f436e910640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436810d0d0 0x7f436819e0a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f436000eaa0 tx=0x7f436000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f435f7fe640 1 -- 172.21.15.111:0/849203626 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436000cdb0 con 0x7f436810d0d0 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436819eb60 con 0x7f436810d0d0 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f435f7fe640 1 -- 172.21.15.111:0/849203626 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f4360005020 con 0x7f436810d0d0 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.224+0000 7f435f7fe640 1 -- 172.21.15.111:0/849203626 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43600116d0 con 0x7f436810d0d0 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.225+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43681a5220 con 0x7f436810d0d0 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.225+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4368104050 con 0x7f436810d0d0 2024-09-16T08:59:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.226+0000 7f435f7fe640 1 -- 172.21.15.111:0/849203626 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f4360004a40 con 0x7f436810d0d0 2024-09-16T08:59:39.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.226+0000 7f435f7fe640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f435407ae10 0x7f435407d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:39.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.227+0000 7f435f7fe640 1 -- 172.21.15.111:0/849203626 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7f436009e5a0 con 0x7f436810d0d0 2024-09-16T08:59:39.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.227+0000 7f436e10f640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f435407ae10 0x7f435407d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:39.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.227+0000 7f436e10f640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f435407ae10 0x7f435407d2d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f43640095f0 tx=0x7f4364005f90 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:39.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.231+0000 7f435f7fe640 1 -- 172.21.15.111:0/849203626 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f4360016030 con 0x7f436810d0d0 2024-09-16T08:59:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.415+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f43681080b0 con 0x7f436810d0d0 2024-09-16T08:59:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.415+0000 7f435f7fe640 1 -- 172.21.15.111:0/849203626 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4360064e20 con 0x7f436810d0d0 2024-09-16T08:59:39.416 INFO:teuthology.orchestra.run.smithi111.stdout:55834574895 2024-09-16T08:59:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f435407ae10 msgr2=0x7f435407d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f435407ae10 0x7f435407d2d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f43640095f0 tx=0x7f4364005f90 comp rx=0 tx=0).stop 2024-09-16T08:59:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436810d0d0 msgr2=0x7f436819e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436810d0d0 0x7f436819e0a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f436000eaa0 tx=0x7f436000ef60 comp rx=0 tx=0).stop 2024-09-16T08:59:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 shutdown_connections 2024-09-16T08:59:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f435407ae10 0x7f435407d2d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f436810d0d0 0x7f436819e0a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4368104590 0x7f4368198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 --2- 172.21.15.111:0/849203626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4368103c70 0x7f43681987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 >> 172.21.15.111:0/849203626 conn(0x7f43680fd6d0 msgr2=0x7f43680fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 shutdown_connections 2024-09-16T08:59:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:39.417+0000 7f4370373640 1 -- 172.21.15.111:0/849203626 wait complete. 2024-09-16T08:59:39.666 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593744 got 184683593745 for osd.6 2024-09-16T08:59:39.666 DEBUG:teuthology.parallel:result is None 2024-09-16T08:59:39.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:39 smithi111 ceph-mon[29803]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-16T08:59:39.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:39 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/849203626' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-16T08:59:39.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:39 smithi111 ceph-mon[40056]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-16T08:59:39.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:39 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/849203626' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-16T08:59:40.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:39 smithi114 ceph-mon[33315]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-16T08:59:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:39 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/849203626' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-16T08:59:40.477 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574895 for osd.1 2024-09-16T08:59:40.477 DEBUG:teuthology.parallel:result is None 2024-09-16T08:59:40.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.703+0000 7f303eb2a640 1 -- 172.21.15.111:0/78248947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30381045d0 msgr2=0x7f303810cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:40.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.703+0000 7f303eb2a640 1 --2- 172.21.15.111:0/78248947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30381045d0 0x7f303810cac0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f302c009900 tx=0x7f302c02a410 comp rx=0 tx=0).stop 2024-09-16T08:59:40.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.703+0000 7f303eb2a640 1 -- 172.21.15.111:0/78248947 shutdown_connections 2024-09-16T08:59:40.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.703+0000 7f303eb2a640 1 --2- 172.21.15.111:0/78248947 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f303810d110 0x7f303810f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.703+0000 7f303eb2a640 1 --2- 172.21.15.111:0/78248947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30381045d0 0x7f303810cac0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.703+0000 7f303eb2a640 1 --2- 172.21.15.111:0/78248947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3038103cb0 0x7f3038104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.703+0000 7f303eb2a640 1 -- 172.21.15.111:0/78248947 >> 172.21.15.111:0/78248947 conn(0x7f30380fd710 msgr2=0x7f30380ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.703+0000 7f303eb2a640 1 -- 172.21.15.111:0/78248947 shutdown_connections 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.703+0000 7f303eb2a640 1 -- 172.21.15.111:0/78248947 wait complete. 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303eb2a640 1 Processor -- start 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303eb2a640 1 -- start start 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303eb2a640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3038103cb0 0x7f3038198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303eb2a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30381045d0 0x7f3038198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303eb2a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f303810d110 0x7f303819e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303eb2a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3038112250 con 0x7f30381045d0 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303eb2a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f30381120d0 con 0x7f303810d110 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303eb2a640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f30381123d0 con 0x7f3038103cb0 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303c8c6640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3038103cb0 0x7f3038198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:40.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303c8c6640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3038103cb0 0x7f3038198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:58500/0 (socket says 172.21.15.111:58500) 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f303c8c6640 1 -- 172.21.15.111:0/3003330982 learned_addr learned my addr 172.21.15.111:0/3003330982 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.704+0000 7f3037fff640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30381045d0 0x7f3038198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303d0c7640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f303810d110 0x7f303819e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303c8c6640 1 -- 172.21.15.111:0/3003330982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f303810d110 msgr2=0x7f303819e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303c8c6640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f303810d110 0x7f303819e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303c8c6640 1 -- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30381045d0 msgr2=0x7f3038198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303c8c6640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30381045d0 0x7f3038198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303c8c6640 1 -- 172.21.15.111:0/3003330982 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303819e8f0 con 0x7f3038103cb0 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303d0c7640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f303810d110 0x7f303819e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303c8c6640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3038103cb0 0x7f3038198840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f302800d950 tx=0x7f302800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f3035ffb640 1 -- 172.21.15.111:0/3003330982 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3028017070 con 0x7f3038103cb0 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f3035ffb640 1 -- 172.21.15.111:0/3003330982 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f3028009c30 con 0x7f3038103cb0 2024-09-16T08:59:40.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303819ebe0 con 0x7f3038103cb0 2024-09-16T08:59:40.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f3035ffb640 1 -- 172.21.15.111:0/3003330982 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302800f3c0 con 0x7f3038103cb0 2024-09-16T08:59:40.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.705+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f30381a52a0 con 0x7f3038103cb0 2024-09-16T08:59:40.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.706+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3038104090 con 0x7f3038103cb0 2024-09-16T08:59:40.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.707+0000 7f3035ffb640 1 -- 172.21.15.111:0/3003330982 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f3028010460 con 0x7f3038103cb0 2024-09-16T08:59:40.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.707+0000 7f3035ffb640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f302007ae10 0x7f302007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:40.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.707+0000 7f3035ffb640 1 -- 172.21.15.111:0/3003330982 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7f302809e030 con 0x7f3038103cb0 2024-09-16T08:59:40.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.707+0000 7f3037fff640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f302007ae10 0x7f302007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:40.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.708+0000 7f3037fff640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f302007ae10 0x7f302007d2d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f302c0057c0 tx=0x7f302c000bc0 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:40.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.712+0000 7f3035ffb640 1 -- 172.21.15.111:0/3003330982 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f30280692e0 con 0x7f3038103cb0 2024-09-16T08:59:40.910 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.909+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3038108180 con 0x7f3038103cb0 2024-09-16T08:59:40.910 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.909+0000 7f3035ffb640 1 -- 172.21.15.111:0/3003330982 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f302806e190 con 0x7f3038103cb0 2024-09-16T08:59:40.910 INFO:teuthology.orchestra.run.smithi111.stdout:158913789974 2024-09-16T08:59:40.912 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.911+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f302007ae10 msgr2=0x7f302007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:40.912 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.911+0000 7f303eb2a640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f302007ae10 0x7f302007d2d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f302c0057c0 tx=0x7f302c000bc0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.912 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.911+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3038103cb0 msgr2=0x7f3038198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:40.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.911+0000 7f303eb2a640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3038103cb0 0x7f3038198840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f302800d950 tx=0x7f302800de10 comp rx=0 tx=0).stop 2024-09-16T08:59:40.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.912+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 shutdown_connections 2024-09-16T08:59:40.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.912+0000 7f303eb2a640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f302007ae10 0x7f302007d2d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.912+0000 7f303eb2a640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f303810d110 0x7f303819e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.912+0000 7f303eb2a640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30381045d0 0x7f3038198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.912+0000 7f303eb2a640 1 --2- 172.21.15.111:0/3003330982 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3038103cb0 0x7f3038198840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:40.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.912+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 >> 172.21.15.111:0/3003330982 conn(0x7f30380fd710 msgr2=0x7f30380fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:40.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.912+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 shutdown_connections 2024-09-16T08:59:40.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:40.912+0000 7f303eb2a640 1 -- 172.21.15.111:0/3003330982 wait complete. 2024-09-16T08:59:41.593 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789972 got 158913789974 for osd.5 2024-09-16T08:59:41.593 DEBUG:teuthology.parallel:result is None 2024-09-16T08:59:41.593 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T08:59:41.593 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T08:59:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:41.887 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T08:59:41.888 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-16T08:59:41.888 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:41.889 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:41.889 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3003330982' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-16T08:59:41.889 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:41.889 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.161 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.161 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.161 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.161 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.161 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.161 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:59:42.161 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3003330982' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:59:42.162 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:59:42.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-16T08:59:42.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3003330982' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-16T08:59:42.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T08:59:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T08:59:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T08:59:42.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.416+0000 7f2139b3e640 1 -- 172.21.15.111:0/2637791592 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f213410d0d0 msgr2=0x7f213410f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:42.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.416+0000 7f2139b3e640 1 --2- 172.21.15.111:0/2637791592 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f213410d0d0 0x7f213410f530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f212c009ca0 tx=0x7f212c02a8c0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.416+0000 7f2139b3e640 1 -- 172.21.15.111:0/2637791592 shutdown_connections 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.416+0000 7f2139b3e640 1 --2- 172.21.15.111:0/2637791592 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f213410d0d0 0x7f213410f530 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.416+0000 7f2139b3e640 1 --2- 172.21.15.111:0/2637791592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2134104590 0x7f213410ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.416+0000 7f2139b3e640 1 --2- 172.21.15.111:0/2637791592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2134103c70 0x7f2134104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.416+0000 7f2139b3e640 1 -- 172.21.15.111:0/2637791592 >> 172.21.15.111:0/2637791592 conn(0x7f21340fd6d0 msgr2=0x7f21340ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.416+0000 7f2139b3e640 1 -- 172.21.15.111:0/2637791592 shutdown_connections 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.416+0000 7f2139b3e640 1 -- 172.21.15.111:0/2637791592 wait complete. 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f2139b3e640 1 Processor -- start 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f2139b3e640 1 -- start start 2024-09-16T08:59:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f2139b3e640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2134103c70 0x7f213419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f2139b3e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2134104590 0x7f213419aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f2139b3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f213410d0d0 0x7f21341a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f2139b3e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2134112210 con 0x7f2134104590 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f2139b3e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2134112090 con 0x7f213410d0d0 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f2139b3e640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f2134112390 con 0x7f2134103c70 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f21337fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2134103c70 0x7f213419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f212bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2134104590 0x7f213419aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f212bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2134104590 0x7f213419aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33150/0 (socket says 172.21.15.111:33150) 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f212bfff640 1 -- 172.21.15.111:0/478626216 learned_addr learned my addr 172.21.15.111:0/478626216 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f21337fe640 1 -- 172.21.15.111:0/478626216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f213410d0d0 msgr2=0x7f21341a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f21337fe640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f213410d0d0 0x7f21341a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f21337fe640 1 -- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2134104590 msgr2=0x7f213419aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f21337fe640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2134104590 0x7f213419aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f21337fe640 1 -- 172.21.15.111:0/478626216 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21341a0a10 con 0x7f2134103c70 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.417+0000 7f21337fe640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2134103c70 0x7f213419a960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f212400d6d0 tx=0x7f212400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.418+0000 7f21317fa640 1 -- 172.21.15.111:0/478626216 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2124017070 con 0x7f2134103c70 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.418+0000 7f21317fa640 1 -- 172.21.15.111:0/478626216 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f2124009820 con 0x7f2134103c70 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.418+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21341a0d00 con 0x7f2134103c70 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.418+0000 7f21317fa640 1 -- 172.21.15.111:0/478626216 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21240049e0 con 0x7f2134103c70 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.418+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f21341a73c0 con 0x7f2134103c70 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.419+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2134104050 con 0x7f2134103c70 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.419+0000 7f21317fa640 1 -- 172.21.15.111:0/478626216 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f2124005220 con 0x7f2134103c70 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.420+0000 7f21317fa640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f212007ae10 0x7f212007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.420+0000 7f21317fa640 1 -- 172.21.15.111:0/478626216 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7f212409dec0 con 0x7f2134103c70 2024-09-16T08:59:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.420+0000 7f212bfff640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f212007ae10 0x7f212007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:42.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.420+0000 7f212bfff640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f212007ae10 0x7f212007d2d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f213410f5f0 tx=0x7f211c00b360 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:42.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.424+0000 7f21317fa640 1 -- 172.21.15.111:0/478626216 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f2124069170 con 0x7f2134103c70 2024-09-16T08:59:42.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.611+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 --> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2134103010 con 0x7f212007ae10 2024-09-16T08:59:42.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.617+0000 7f21317fa640 1 -- 172.21.15.111:0/478626216 <== mgr.14607 v2:172.21.15.111:6800/1888355313 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346435 (secure 0 0 0) 0x7f2134103010 con 0x7f212007ae10 2024-09-16T08:59:42.621 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-09-16T08:59:42.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f212007ae10 msgr2=0x7f212007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:42.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f212007ae10 0x7f212007d2d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f213410f5f0 tx=0x7f211c00b360 comp rx=0 tx=0).stop 2024-09-16T08:59:42.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2134103c70 msgr2=0x7f213419a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:42.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2134103c70 0x7f213419a960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f212400d6d0 tx=0x7f212400db90 comp rx=0 tx=0).stop 2024-09-16T08:59:42.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 shutdown_connections 2024-09-16T08:59:42.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f212007ae10 0x7f212007d2d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f213410d0d0 0x7f21341a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2134104590 0x7f213419aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 --2- 172.21.15.111:0/478626216 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2134103c70 0x7f213419a960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 >> 172.21.15.111:0/478626216 conn(0x7f21340fd6d0 msgr2=0x7f21340ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 shutdown_connections 2024-09-16T08:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:42.620+0000 7f2139b3e640 1 -- 172.21.15.111:0/478626216 wait complete. 2024-09-16T08:59:43.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:08:59:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T08:59:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T08:59:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi111 to 3760M 2024-09-16T08:59:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Updating smithi111:/etc/ceph/ceph.conf 2024-09-16T08:59:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-16T08:59:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:59:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[29803]: Deploying daemon alertmanager.a on smithi111 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi111 to 3760M 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Updating smithi111:/etc/ceph/ceph.conf 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:43 smithi111 ceph-mon[40056]: Deploying daemon alertmanager.a on smithi111 2024-09-16T08:59:43.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi111 to 3760M 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Updating smithi111:/etc/ceph/ceph.conf 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Updating smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Updating smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:43 smithi114 ceph-mon[33315]: Deploying daemon alertmanager.a on smithi111 2024-09-16T08:59:43.872 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-09-16T08:59:42.453667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":885,"num_read_kb":742,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":235956,"kb_used_data":7260,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725260,"statfs":{"total":767960285184,"available":767718666240,"internally_reserved":0,"allocated":7434240,"data_stored":4068817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":186,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.998426"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204641+0000","last_change":"2024-09-16T08:58:59.257450+0000","last_active":"2024-09-16T08:59:31.204641+0000","last_peered":"2024-09-16T08:59:31.204641+0000","last_clean":"2024-09-16T08:59:31.204641+0000","last_became_active":"2024-09-16T08:58:59.257177+0000","last_became_peered":"2024-09-16T08:58:59.257177+0000","last_unstale":"2024-09-16T08:59:31.204641+0000","last_undegraded":"2024-09-16T08:59:31.204641+0000","last_fullsized":"2024-09-16T08:59:31.204641+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:08:16.833508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203163+0000","last_change":"2024-09-16T08:59:01.266150+0000","last_active":"2024-09-16T08:59:31.203163+0000","last_peered":"2024-09-16T08:59:31.203163+0000","last_clean":"2024-09-16T08:59:31.203163+0000","last_became_active":"2024-09-16T08:59:01.265521+0000","last_became_peered":"2024-09-16T08:59:01.265521+0000","last_unstale":"2024-09-16T08:59:31.203163+0000","last_undegraded":"2024-09-16T08:59:31.203163+0000","last_fullsized":"2024-09-16T08:59:31.203163+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:28.698635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439081+0000","last_change":"2024-09-16T08:59:03.284173+0000","last_active":"2024-09-16T08:59:30.439081+0000","last_peered":"2024-09-16T08:59:30.439081+0000","last_clean":"2024-09-16T08:59:30.439081+0000","last_became_active":"2024-09-16T08:59:03.283877+0000","last_became_peered":"2024-09-16T08:59:03.283877+0000","last_unstale":"2024-09-16T08:59:30.439081+0000","last_undegraded":"2024-09-16T08:59:30.439081+0000","last_fullsized":"2024-09-16T08:59:30.439081+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:53.510657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203491+0000","last_change":"2024-09-16T08:59:05.289579+0000","last_active":"2024-09-16T08:59:31.203491+0000","last_peered":"2024-09-16T08:59:31.203491+0000","last_clean":"2024-09-16T08:59:31.203491+0000","last_became_active":"2024-09-16T08:59:05.289266+0000","last_became_peered":"2024-09-16T08:59:05.289266+0000","last_unstale":"2024-09-16T08:59:31.203491+0000","last_undegraded":"2024-09-16T08:59:31.203491+0000","last_fullsized":"2024-09-16T08:59:31.203491+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:01:50.876283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440642+0000","last_change":"2024-09-16T08:59:05.291614+0000","last_active":"2024-09-16T08:59:30.440642+0000","last_peered":"2024-09-16T08:59:30.440642+0000","last_clean":"2024-09-16T08:59:30.440642+0000","last_became_active":"2024-09-16T08:59:05.291295+0000","last_became_peered":"2024-09-16T08:59:05.291295+0000","last_unstale":"2024-09-16T08:59:30.440642+0000","last_undegraded":"2024-09-16T08:59:30.440642+0000","last_fullsized":"2024-09-16T08:59:30.440642+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:44:58.709337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441188+0000","last_change":"2024-09-16T08:58:59.262723+0000","last_active":"2024-09-16T08:59:30.441188+0000","last_peered":"2024-09-16T08:59:30.441188+0000","last_clean":"2024-09-16T08:59:30.441188+0000","last_became_active":"2024-09-16T08:58:59.262427+0000","last_became_peered":"2024-09-16T08:58:59.262427+0000","last_unstale":"2024-09-16T08:59:30.441188+0000","last_undegraded":"2024-09-16T08:59:30.441188+0000","last_fullsized":"2024-09-16T08:59:30.441188+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:33:20.565876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"61'15","reported_seq":51,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440436+0000","last_change":"2024-09-16T08:59:01.272595+0000","last_active":"2024-09-16T08:59:30.440436+0000","last_peered":"2024-09-16T08:59:30.440436+0000","last_clean":"2024-09-16T08:59:30.440436+0000","last_became_active":"2024-09-16T08:59:01.272304+0000","last_became_peered":"2024-09-16T08:59:01.272304+0000","last_unstale":"2024-09-16T08:59:30.440436+0000","last_undegraded":"2024-09-16T08:59:30.440436+0000","last_fullsized":"2024-09-16T08:59:30.440436+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:55:10.089748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203058+0000","last_change":"2024-09-16T08:59:03.285713+0000","last_active":"2024-09-16T08:59:31.203058+0000","last_peered":"2024-09-16T08:59:31.203058+0000","last_clean":"2024-09-16T08:59:31.203058+0000","last_became_active":"2024-09-16T08:59:03.285390+0000","last_became_peered":"2024-09-16T08:59:03.285390+0000","last_unstale":"2024-09-16T08:59:31.203058+0000","last_undegraded":"2024-09-16T08:59:31.203058+0000","last_fullsized":"2024-09-16T08:59:31.203058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:51.755516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775907+0000","last_change":"2024-09-16T08:58:59.263378+0000","last_active":"2024-09-16T08:59:30.775907+0000","last_peered":"2024-09-16T08:59:30.775907+0000","last_clean":"2024-09-16T08:59:30.775907+0000","last_became_active":"2024-09-16T08:58:59.263094+0000","last_became_peered":"2024-09-16T08:58:59.263094+0000","last_unstale":"2024-09-16T08:59:30.775907+0000","last_undegraded":"2024-09-16T08:59:30.775907+0000","last_fullsized":"2024-09-16T08:59:30.775907+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:15.702764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203704+0000","last_change":"2024-09-16T08:59:01.266657+0000","last_active":"2024-09-16T08:59:31.203704+0000","last_peered":"2024-09-16T08:59:31.203704+0000","last_clean":"2024-09-16T08:59:31.203704+0000","last_became_active":"2024-09-16T08:59:01.266349+0000","last_became_peered":"2024-09-16T08:59:01.266349+0000","last_unstale":"2024-09-16T08:59:31.203704+0000","last_undegraded":"2024-09-16T08:59:31.203704+0000","last_fullsized":"2024-09-16T08:59:31.203704+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:11.945878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775547+0000","last_change":"2024-09-16T08:59:03.281873+0000","last_active":"2024-09-16T08:59:30.775547+0000","last_peered":"2024-09-16T08:59:30.775547+0000","last_clean":"2024-09-16T08:59:30.775547+0000","last_became_active":"2024-09-16T08:59:03.281541+0000","last_became_peered":"2024-09-16T08:59:03.281541+0000","last_unstale":"2024-09-16T08:59:30.775547+0000","last_undegraded":"2024-09-16T08:59:30.775547+0000","last_fullsized":"2024-09-16T08:59:30.775547+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:04:02.903335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203263+0000","last_change":"2024-09-16T08:59:05.293990+0000","last_active":"2024-09-16T08:59:31.203263+0000","last_peered":"2024-09-16T08:59:31.203263+0000","last_clean":"2024-09-16T08:59:31.203263+0000","last_became_active":"2024-09-16T08:59:05.293648+0000","last_became_peered":"2024-09-16T08:59:05.293648+0000","last_unstale":"2024-09-16T08:59:31.203263+0000","last_undegraded":"2024-09-16T08:59:31.203263+0000","last_fullsized":"2024-09-16T08:59:31.203263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:10:42.342067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775773+0000","last_change":"2024-09-16T08:58:59.260319+0000","last_active":"2024-09-16T08:59:30.775773+0000","last_peered":"2024-09-16T08:59:30.775773+0000","last_clean":"2024-09-16T08:59:30.775773+0000","last_became_active":"2024-09-16T08:58:59.260043+0000","last_became_peered":"2024-09-16T08:58:59.260043+0000","last_unstale":"2024-09-16T08:59:30.775773+0000","last_undegraded":"2024-09-16T08:59:30.775773+0000","last_fullsized":"2024-09-16T08:59:30.775773+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:11.800206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"61'5","reported_seq":36,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203050+0000","last_change":"2024-09-16T08:59:01.266539+0000","last_active":"2024-09-16T08:59:31.203050+0000","last_peered":"2024-09-16T08:59:31.203050+0000","last_clean":"2024-09-16T08:59:31.203050+0000","last_became_active":"2024-09-16T08:59:01.265489+0000","last_became_peered":"2024-09-16T08:59:01.265489+0000","last_unstale":"2024-09-16T08:59:31.203050+0000","last_undegraded":"2024-09-16T08:59:31.203050+0000","last_fullsized":"2024-09-16T08:59:31.203050+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:07.428651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440512+0000","last_change":"2024-09-16T08:59:03.286289+0000","last_active":"2024-09-16T08:59:30.440512+0000","last_peered":"2024-09-16T08:59:30.440512+0000","last_clean":"2024-09-16T08:59:30.440512+0000","last_became_active":"2024-09-16T08:59:03.285037+0000","last_became_peered":"2024-09-16T08:59:03.285037+0000","last_unstale":"2024-09-16T08:59:30.440512+0000","last_undegraded":"2024-09-16T08:59:30.440512+0000","last_fullsized":"2024-09-16T08:59:30.440512+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:35.283984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775404+0000","last_change":"2024-09-16T08:59:05.292585+0000","last_active":"2024-09-16T08:59:30.775404+0000","last_peered":"2024-09-16T08:59:30.775404+0000","last_clean":"2024-09-16T08:59:30.775404+0000","last_became_active":"2024-09-16T08:59:05.291469+0000","last_became_peered":"2024-09-16T08:59:05.291469+0000","last_unstale":"2024-09-16T08:59:30.775404+0000","last_undegraded":"2024-09-16T08:59:30.775404+0000","last_fullsized":"2024-09-16T08:59:30.775404+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:43:12.870803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204212+0000","last_change":"2024-09-16T08:59:05.287502+0000","last_active":"2024-09-16T08:59:31.204212+0000","last_peered":"2024-09-16T08:59:31.204212+0000","last_clean":"2024-09-16T08:59:31.204212+0000","last_became_active":"2024-09-16T08:59:05.287231+0000","last_became_peered":"2024-09-16T08:59:05.287231+0000","last_unstale":"2024-09-16T08:59:31.204212+0000","last_undegraded":"2024-09-16T08:59:31.204212+0000","last_fullsized":"2024-09-16T08:59:31.204212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:15.074348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204895+0000","last_change":"2024-09-16T08:58:59.260923+0000","last_active":"2024-09-16T08:59:31.204895+0000","last_peered":"2024-09-16T08:59:31.204895+0000","last_clean":"2024-09-16T08:59:31.204895+0000","last_became_active":"2024-09-16T08:58:59.260638+0000","last_became_peered":"2024-09-16T08:58:59.260638+0000","last_unstale":"2024-09-16T08:59:31.204895+0000","last_undegraded":"2024-09-16T08:59:31.204895+0000","last_fullsized":"2024-09-16T08:59:31.204895+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:49.994022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"61'15","reported_seq":51,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781962+0000","last_change":"2024-09-16T08:59:01.271728+0000","last_active":"2024-09-16T08:59:30.781962+0000","last_peered":"2024-09-16T08:59:30.781962+0000","last_clean":"2024-09-16T08:59:30.781962+0000","last_became_active":"2024-09-16T08:59:01.271428+0000","last_became_peered":"2024-09-16T08:59:01.271428+0000","last_unstale":"2024-09-16T08:59:30.781962+0000","last_undegraded":"2024-09-16T08:59:30.781962+0000","last_fullsized":"2024-09-16T08:59:30.781962+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:57.801373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440183+0000","last_change":"2024-09-16T08:59:03.284033+0000","last_active":"2024-09-16T08:59:30.440183+0000","last_peered":"2024-09-16T08:59:30.440183+0000","last_clean":"2024-09-16T08:59:30.440183+0000","last_became_active":"2024-09-16T08:59:03.283603+0000","last_became_peered":"2024-09-16T08:59:03.283603+0000","last_unstale":"2024-09-16T08:59:30.440183+0000","last_undegraded":"2024-09-16T08:59:30.440183+0000","last_fullsized":"2024-09-16T08:59:30.440183+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:58:49.047897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441203+0000","last_change":"2024-09-16T08:59:05.297491+0000","last_active":"2024-09-16T08:59:30.441203+0000","last_peered":"2024-09-16T08:59:30.441203+0000","last_clean":"2024-09-16T08:59:30.441203+0000","last_became_active":"2024-09-16T08:59:05.297022+0000","last_became_peered":"2024-09-16T08:59:05.297022+0000","last_unstale":"2024-09-16T08:59:30.441203+0000","last_undegraded":"2024-09-16T08:59:30.441203+0000","last_fullsized":"2024-09-16T08:59:30.441203+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:34:04.684922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"54'1","reported_seq":36,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203823+0000","last_change":"2024-09-16T08:58:59.256620+0000","last_active":"2024-09-16T08:59:31.203823+0000","last_peered":"2024-09-16T08:59:31.203823+0000","last_clean":"2024-09-16T08:59:31.203823+0000","last_became_active":"2024-09-16T08:58:59.256268+0000","last_became_peered":"2024-09-16T08:58:59.256268+0000","last_unstale":"2024-09-16T08:59:31.203823+0000","last_undegraded":"2024-09-16T08:59:31.203823+0000","last_fullsized":"2024-09-16T08:59:31.203823+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:10.992783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"61'12","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440866+0000","last_change":"2024-09-16T08:59:01.269703+0000","last_active":"2024-09-16T08:59:30.440866+0000","last_peered":"2024-09-16T08:59:30.440866+0000","last_clean":"2024-09-16T08:59:30.440866+0000","last_became_active":"2024-09-16T08:59:01.269373+0000","last_became_peered":"2024-09-16T08:59:01.269373+0000","last_unstale":"2024-09-16T08:59:30.440866+0000","last_undegraded":"2024-09-16T08:59:30.440866+0000","last_fullsized":"2024-09-16T08:59:30.440866+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:10.213444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203980+0000","last_change":"2024-09-16T08:59:03.273760+0000","last_active":"2024-09-16T08:59:31.203980+0000","last_peered":"2024-09-16T08:59:31.203980+0000","last_clean":"2024-09-16T08:59:31.203980+0000","last_became_active":"2024-09-16T08:59:03.273475+0000","last_became_peered":"2024-09-16T08:59:03.273475+0000","last_unstale":"2024-09-16T08:59:31.203980+0000","last_undegraded":"2024-09-16T08:59:31.203980+0000","last_fullsized":"2024-09-16T08:59:31.203980+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:42:32.472066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"61'1","reported_seq":19,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440465+0000","last_change":"2024-09-16T08:59:05.295383+0000","last_active":"2024-09-16T08:59:30.440465+0000","last_peered":"2024-09-16T08:59:30.440465+0000","last_clean":"2024-09-16T08:59:30.440465+0000","last_became_active":"2024-09-16T08:59:05.295028+0000","last_became_peered":"2024-09-16T08:59:05.295028+0000","last_unstale":"2024-09-16T08:59:30.440465+0000","last_undegraded":"2024-09-16T08:59:30.440465+0000","last_fullsized":"2024-09-16T08:59:30.440465+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:06:24.957000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"54'1","reported_seq":31,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439792+0000","last_change":"2024-09-16T08:58:59.259592+0000","last_active":"2024-09-16T08:59:30.439792+0000","last_peered":"2024-09-16T08:59:30.439792+0000","last_clean":"2024-09-16T08:59:30.439792+0000","last_became_active":"2024-09-16T08:58:59.259298+0000","last_became_peered":"2024-09-16T08:58:59.259298+0000","last_unstale":"2024-09-16T08:59:30.439792+0000","last_undegraded":"2024-09-16T08:59:30.439792+0000","last_fullsized":"2024-09-16T08:59:30.439792+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:09:13.136494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"61'10","reported_seq":41,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204250+0000","last_change":"2024-09-16T08:59:01.435269+0000","last_active":"2024-09-16T08:59:31.204250+0000","last_peered":"2024-09-16T08:59:31.204250+0000","last_clean":"2024-09-16T08:59:31.204250+0000","last_became_active":"2024-09-16T08:59:01.434961+0000","last_became_peered":"2024-09-16T08:59:01.434961+0000","last_unstale":"2024-09-16T08:59:31.204250+0000","last_undegraded":"2024-09-16T08:59:31.204250+0000","last_fullsized":"2024-09-16T08:59:31.204250+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:03.566813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"61'8","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318219+0000","last_change":"2024-09-16T08:59:03.283645+0000","last_active":"2024-09-16T08:59:34.318219+0000","last_peered":"2024-09-16T08:59:34.318219+0000","last_clean":"2024-09-16T08:59:34.318219+0000","last_became_active":"2024-09-16T08:59:03.282895+0000","last_became_peered":"2024-09-16T08:59:03.282895+0000","last_unstale":"2024-09-16T08:59:34.318219+0000","last_undegraded":"2024-09-16T08:59:34.318219+0000","last_fullsized":"2024-09-16T08:59:34.318219+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:31.346405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"61'15","reported_seq":51,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441525+0000","last_change":"2024-09-16T08:59:01.273582+0000","last_active":"2024-09-16T08:59:30.441525+0000","last_peered":"2024-09-16T08:59:30.441525+0000","last_clean":"2024-09-16T08:59:30.441525+0000","last_became_active":"2024-09-16T08:59:01.273304+0000","last_became_peered":"2024-09-16T08:59:01.273304+0000","last_unstale":"2024-09-16T08:59:30.441525+0000","last_undegraded":"2024-09-16T08:59:30.441525+0000","last_fullsized":"2024-09-16T08:59:30.441525+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:07.272672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440545+0000","last_change":"2024-09-16T08:58:59.258366+0000","last_active":"2024-09-16T08:59:30.440545+0000","last_peered":"2024-09-16T08:59:30.440545+0000","last_clean":"2024-09-16T08:59:30.440545+0000","last_became_active":"2024-09-16T08:58:59.258088+0000","last_became_peered":"2024-09-16T08:58:59.258088+0000","last_unstale":"2024-09-16T08:59:30.440545+0000","last_undegraded":"2024-09-16T08:59:30.440545+0000","last_fullsized":"2024-09-16T08:59:30.440545+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:39:28.370346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318204+0000","last_change":"2024-09-16T08:59:03.278461+0000","last_active":"2024-09-16T08:59:34.318204+0000","last_peered":"2024-09-16T08:59:34.318204+0000","last_clean":"2024-09-16T08:59:34.318204+0000","last_became_active":"2024-09-16T08:59:03.278191+0000","last_became_peered":"2024-09-16T08:59:03.278191+0000","last_unstale":"2024-09-16T08:59:34.318204+0000","last_undegraded":"2024-09-16T08:59:34.318204+0000","last_fullsized":"2024-09-16T08:59:34.318204+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:52:27.612755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775636+0000","last_change":"2024-09-16T08:59:05.292173+0000","last_active":"2024-09-16T08:59:30.775636+0000","last_peered":"2024-09-16T08:59:30.775636+0000","last_clean":"2024-09-16T08:59:30.775636+0000","last_became_active":"2024-09-16T08:59:05.291813+0000","last_became_peered":"2024-09-16T08:59:05.291813+0000","last_unstale":"2024-09-16T08:59:30.775636+0000","last_undegraded":"2024-09-16T08:59:30.775636+0000","last_fullsized":"2024-09-16T08:59:30.775636+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:52:06.061769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"61'18","reported_seq":58,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440221+0000","last_change":"2024-09-16T08:59:01.433916+0000","last_active":"2024-09-16T08:59:30.440221+0000","last_peered":"2024-09-16T08:59:30.440221+0000","last_clean":"2024-09-16T08:59:30.440221+0000","last_became_active":"2024-09-16T08:59:01.433358+0000","last_became_peered":"2024-09-16T08:59:01.433358+0000","last_unstale":"2024-09-16T08:59:30.440221+0000","last_undegraded":"2024-09-16T08:59:30.440221+0000","last_fullsized":"2024-09-16T08:59:30.440221+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:13.471300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441895+0000","last_change":"2024-09-16T08:58:59.264510+0000","last_active":"2024-09-16T08:59:30.441895+0000","last_peered":"2024-09-16T08:59:30.441895+0000","last_clean":"2024-09-16T08:59:30.441895+0000","last_became_active":"2024-09-16T08:58:59.264199+0000","last_became_peered":"2024-09-16T08:58:59.264199+0000","last_unstale":"2024-09-16T08:59:30.441895+0000","last_undegraded":"2024-09-16T08:59:30.441895+0000","last_fullsized":"2024-09-16T08:59:30.441895+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:32.005141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203160+0000","last_change":"2024-09-16T08:59:03.280701+0000","last_active":"2024-09-16T08:59:31.203160+0000","last_peered":"2024-09-16T08:59:31.203160+0000","last_clean":"2024-09-16T08:59:31.203160+0000","last_became_active":"2024-09-16T08:59:03.280440+0000","last_became_peered":"2024-09-16T08:59:03.280440+0000","last_unstale":"2024-09-16T08:59:31.203160+0000","last_undegraded":"2024-09-16T08:59:31.203160+0000","last_fullsized":"2024-09-16T08:59:31.203160+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:21:52.705170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203681+0000","last_change":"2024-09-16T08:59:05.288877+0000","last_active":"2024-09-16T08:59:31.203681+0000","last_peered":"2024-09-16T08:59:31.203681+0000","last_clean":"2024-09-16T08:59:31.203681+0000","last_became_active":"2024-09-16T08:59:05.288570+0000","last_became_peered":"2024-09-16T08:59:05.288570+0000","last_unstale":"2024-09-16T08:59:31.203681+0000","last_undegraded":"2024-09-16T08:59:31.203681+0000","last_fullsized":"2024-09-16T08:59:31.203681+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:59.393078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"61'14","reported_seq":47,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203368+0000","last_change":"2024-09-16T08:59:01.264825+0000","last_active":"2024-09-16T08:59:31.203368+0000","last_peered":"2024-09-16T08:59:31.203368+0000","last_clean":"2024-09-16T08:59:31.203368+0000","last_became_active":"2024-09-16T08:59:01.264064+0000","last_became_peered":"2024-09-16T08:59:01.264064+0000","last_unstale":"2024-09-16T08:59:31.203368+0000","last_undegraded":"2024-09-16T08:59:31.203368+0000","last_fullsized":"2024-09-16T08:59:31.203368+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:23.056600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"54'1","reported_seq":31,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204793+0000","last_change":"2024-09-16T08:58:59.259100+0000","last_active":"2024-09-16T08:59:31.204793+0000","last_peered":"2024-09-16T08:59:31.204793+0000","last_clean":"2024-09-16T08:59:31.204793+0000","last_became_active":"2024-09-16T08:58:59.258795+0000","last_became_peered":"2024-09-16T08:58:59.258795+0000","last_unstale":"2024-09-16T08:59:31.204793+0000","last_undegraded":"2024-09-16T08:59:31.204793+0000","last_fullsized":"2024-09-16T08:59:31.204793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:39.926579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318349+0000","last_change":"2024-09-16T08:59:03.280018+0000","last_active":"2024-09-16T08:59:34.318349+0000","last_peered":"2024-09-16T08:59:34.318349+0000","last_clean":"2024-09-16T08:59:34.318349+0000","last_became_active":"2024-09-16T08:59:03.279718+0000","last_became_peered":"2024-09-16T08:59:03.279718+0000","last_unstale":"2024-09-16T08:59:34.318349+0000","last_undegraded":"2024-09-16T08:59:34.318349+0000","last_fullsized":"2024-09-16T08:59:34.318349+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:37.177323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440543+0000","last_change":"2024-09-16T08:59:05.295105+0000","last_active":"2024-09-16T08:59:30.440543+0000","last_peered":"2024-09-16T08:59:30.440543+0000","last_clean":"2024-09-16T08:59:30.440543+0000","last_became_active":"2024-09-16T08:59:05.294831+0000","last_became_peered":"2024-09-16T08:59:05.294831+0000","last_unstale":"2024-09-16T08:59:30.440543+0000","last_undegraded":"2024-09-16T08:59:30.440543+0000","last_fullsized":"2024-09-16T08:59:30.440543+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:36:03.315109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"61'10","reported_seq":41,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439429+0000","last_change":"2024-09-16T08:59:01.267284+0000","last_active":"2024-09-16T08:59:30.439429+0000","last_peered":"2024-09-16T08:59:30.439429+0000","last_clean":"2024-09-16T08:59:30.439429+0000","last_became_active":"2024-09-16T08:59:01.266969+0000","last_became_peered":"2024-09-16T08:59:01.266969+0000","last_unstale":"2024-09-16T08:59:30.439429+0000","last_undegraded":"2024-09-16T08:59:30.439429+0000","last_fullsized":"2024-09-16T08:59:30.439429+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:55:10.837684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775949+0000","last_change":"2024-09-16T08:58:59.256998+0000","last_active":"2024-09-16T08:59:30.775949+0000","last_peered":"2024-09-16T08:59:30.775949+0000","last_clean":"2024-09-16T08:59:30.775949+0000","last_became_active":"2024-09-16T08:58:59.256705+0000","last_became_peered":"2024-09-16T08:58:59.256705+0000","last_unstale":"2024-09-16T08:59:30.775949+0000","last_undegraded":"2024-09-16T08:59:30.775949+0000","last_fullsized":"2024-09-16T08:59:30.775949+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:14:43.704274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"61'8","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318266+0000","last_change":"2024-09-16T08:59:03.286630+0000","last_active":"2024-09-16T08:59:34.318266+0000","last_peered":"2024-09-16T08:59:34.318266+0000","last_clean":"2024-09-16T08:59:34.318266+0000","last_became_active":"2024-09-16T08:59:03.286357+0000","last_became_peered":"2024-09-16T08:59:03.286357+0000","last_unstale":"2024-09-16T08:59:34.318266+0000","last_undegraded":"2024-09-16T08:59:34.318266+0000","last_fullsized":"2024-09-16T08:59:34.318266+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:07:29.240876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203414+0000","last_change":"2024-09-16T08:59:05.289940+0000","last_active":"2024-09-16T08:59:31.203414+0000","last_peered":"2024-09-16T08:59:31.203414+0000","last_clean":"2024-09-16T08:59:31.203414+0000","last_became_active":"2024-09-16T08:59:05.289652+0000","last_became_peered":"2024-09-16T08:59:05.289652+0000","last_unstale":"2024-09-16T08:59:31.203414+0000","last_undegraded":"2024-09-16T08:59:31.203414+0000","last_fullsized":"2024-09-16T08:59:31.203414+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:59.679662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"61'19","reported_seq":62,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204625+0000","last_change":"2024-09-16T08:59:01.433867+0000","last_active":"2024-09-16T08:59:31.204625+0000","last_peered":"2024-09-16T08:59:31.204625+0000","last_clean":"2024-09-16T08:59:31.204625+0000","last_became_active":"2024-09-16T08:59:01.433528+0000","last_became_peered":"2024-09-16T08:59:01.433528+0000","last_unstale":"2024-09-16T08:59:31.204625+0000","last_undegraded":"2024-09-16T08:59:31.204625+0000","last_fullsized":"2024-09-16T08:59:31.204625+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:25.700392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439917+0000","last_change":"2024-09-16T08:58:59.258343+0000","last_active":"2024-09-16T08:59:30.439917+0000","last_peered":"2024-09-16T08:59:30.439917+0000","last_clean":"2024-09-16T08:59:30.439917+0000","last_became_active":"2024-09-16T08:58:59.258097+0000","last_became_peered":"2024-09-16T08:58:59.258097+0000","last_unstale":"2024-09-16T08:59:30.439917+0000","last_undegraded":"2024-09-16T08:59:30.439917+0000","last_fullsized":"2024-09-16T08:59:30.439917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:41.283169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203069+0000","last_change":"2024-09-16T08:59:03.283011+0000","last_active":"2024-09-16T08:59:31.203069+0000","last_peered":"2024-09-16T08:59:31.203069+0000","last_clean":"2024-09-16T08:59:31.203069+0000","last_became_active":"2024-09-16T08:59:03.282735+0000","last_became_peered":"2024-09-16T08:59:03.282735+0000","last_unstale":"2024-09-16T08:59:31.203069+0000","last_undegraded":"2024-09-16T08:59:31.203069+0000","last_fullsized":"2024-09-16T08:59:31.203069+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:35.440091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439039+0000","last_change":"2024-09-16T08:59:05.285275+0000","last_active":"2024-09-16T08:59:30.439039+0000","last_peered":"2024-09-16T08:59:30.439039+0000","last_clean":"2024-09-16T08:59:30.439039+0000","last_became_active":"2024-09-16T08:59:05.284926+0000","last_became_peered":"2024-09-16T08:59:05.284926+0000","last_unstale":"2024-09-16T08:59:30.439039+0000","last_undegraded":"2024-09-16T08:59:30.439039+0000","last_fullsized":"2024-09-16T08:59:30.439039+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:38.799825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"61'28","reported_seq":83,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318340+0000","last_change":"2024-09-16T08:59:01.273742+0000","last_active":"2024-09-16T08:59:34.318340+0000","last_peered":"2024-09-16T08:59:34.318340+0000","last_clean":"2024-09-16T08:59:34.318340+0000","last_became_active":"2024-09-16T08:59:01.273463+0000","last_became_peered":"2024-09-16T08:59:01.273463+0000","last_unstale":"2024-09-16T08:59:34.318340+0000","last_undegraded":"2024-09-16T08:59:34.318340+0000","last_fullsized":"2024-09-16T08:59:34.318340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:45:46.472915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204050+0000","last_change":"2024-09-16T08:58:59.264820+0000","last_active":"2024-09-16T08:59:31.204050+0000","last_peered":"2024-09-16T08:59:31.204050+0000","last_clean":"2024-09-16T08:59:31.204050+0000","last_became_active":"2024-09-16T08:58:59.264413+0000","last_became_peered":"2024-09-16T08:58:59.264413+0000","last_unstale":"2024-09-16T08:59:31.204050+0000","last_undegraded":"2024-09-16T08:59:31.204050+0000","last_fullsized":"2024-09-16T08:59:31.204050+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:43:27.553191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":40,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775624+0000","last_change":"2024-09-16T08:59:01.262367+0000","last_active":"2024-09-16T08:59:30.775624+0000","last_peered":"2024-09-16T08:59:30.775624+0000","last_clean":"2024-09-16T08:59:30.775624+0000","last_became_active":"2024-09-16T08:58:58.246232+0000","last_became_peered":"2024-09-16T08:58:58.246232+0000","last_unstale":"2024-09-16T08:59:30.775624+0000","last_undegraded":"2024-09-16T08:59:30.775624+0000","last_fullsized":"2024-09-16T08:59:30.775624+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_clean_scrub_stamp":"2024-09-16T08:58:57.230240+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:29.095207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001635145,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204290+0000","last_change":"2024-09-16T08:59:03.285616+0000","last_active":"2024-09-16T08:59:31.204290+0000","last_peered":"2024-09-16T08:59:31.204290+0000","last_clean":"2024-09-16T08:59:31.204290+0000","last_became_active":"2024-09-16T08:59:03.284205+0000","last_became_peered":"2024-09-16T08:59:03.284205+0000","last_unstale":"2024-09-16T08:59:31.204290+0000","last_undegraded":"2024-09-16T08:59:31.204290+0000","last_fullsized":"2024-09-16T08:59:31.204290+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:18:00.305433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440910+0000","last_change":"2024-09-16T08:59:05.293367+0000","last_active":"2024-09-16T08:59:30.440910+0000","last_peered":"2024-09-16T08:59:30.440910+0000","last_clean":"2024-09-16T08:59:30.440910+0000","last_became_active":"2024-09-16T08:59:05.291800+0000","last_became_peered":"2024-09-16T08:59:05.291800+0000","last_unstale":"2024-09-16T08:59:30.440910+0000","last_undegraded":"2024-09-16T08:59:30.440910+0000","last_fullsized":"2024-09-16T08:59:30.440910+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:49.809698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"61'13","reported_seq":53,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441660+0000","last_change":"2024-09-16T08:59:01.271118+0000","last_active":"2024-09-16T08:59:30.441660+0000","last_peered":"2024-09-16T08:59:30.441660+0000","last_clean":"2024-09-16T08:59:30.441660+0000","last_became_active":"2024-09-16T08:59:01.270835+0000","last_became_peered":"2024-09-16T08:59:01.270835+0000","last_unstale":"2024-09-16T08:59:30.441660+0000","last_undegraded":"2024-09-16T08:59:30.441660+0000","last_fullsized":"2024-09-16T08:59:30.441660+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:30.651659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439285+0000","last_change":"2024-09-16T08:58:59.263658+0000","last_active":"2024-09-16T08:59:30.439285+0000","last_peered":"2024-09-16T08:59:30.439285+0000","last_clean":"2024-09-16T08:59:30.439285+0000","last_became_active":"2024-09-16T08:58:59.263366+0000","last_became_peered":"2024-09-16T08:58:59.263366+0000","last_unstale":"2024-09-16T08:59:30.439285+0000","last_undegraded":"2024-09-16T08:59:30.439285+0000","last_fullsized":"2024-09-16T08:59:30.439285+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:07.710217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'1","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781971+0000","last_change":"2024-09-16T08:59:01.263688+0000","last_active":"2024-09-16T08:59:30.781971+0000","last_peered":"2024-09-16T08:59:30.781971+0000","last_clean":"2024-09-16T08:59:30.781971+0000","last_became_active":"2024-09-16T08:58:58.248552+0000","last_became_peered":"2024-09-16T08:58:58.248552+0000","last_unstale":"2024-09-16T08:59:30.781971+0000","last_undegraded":"2024-09-16T08:59:30.781971+0000","last_fullsized":"2024-09-16T08:59:30.781971+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_clean_scrub_stamp":"2024-09-16T08:58:57.230240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:44.765434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003167675,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781651+0000","last_change":"2024-09-16T08:59:03.280876+0000","last_active":"2024-09-16T08:59:30.781651+0000","last_peered":"2024-09-16T08:59:30.781651+0000","last_clean":"2024-09-16T08:59:30.781651+0000","last_became_active":"2024-09-16T08:59:03.280590+0000","last_became_peered":"2024-09-16T08:59:03.280590+0000","last_unstale":"2024-09-16T08:59:30.781651+0000","last_undegraded":"2024-09-16T08:59:30.781651+0000","last_fullsized":"2024-09-16T08:59:30.781651+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:43.999322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440644+0000","last_change":"2024-09-16T08:59:05.293663+0000","last_active":"2024-09-16T08:59:30.440644+0000","last_peered":"2024-09-16T08:59:30.440644+0000","last_clean":"2024-09-16T08:59:30.440644+0000","last_became_active":"2024-09-16T08:59:05.293366+0000","last_became_peered":"2024-09-16T08:59:05.293366+0000","last_unstale":"2024-09-16T08:59:30.440644+0000","last_undegraded":"2024-09-16T08:59:30.440644+0000","last_fullsized":"2024-09-16T08:59:30.440644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:43.979621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"61'12","reported_seq":44,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204380+0000","last_change":"2024-09-16T08:59:01.272958+0000","last_active":"2024-09-16T08:59:31.204380+0000","last_peered":"2024-09-16T08:59:31.204380+0000","last_clean":"2024-09-16T08:59:31.204380+0000","last_became_active":"2024-09-16T08:59:01.272635+0000","last_became_peered":"2024-09-16T08:59:01.272635+0000","last_unstale":"2024-09-16T08:59:31.204380+0000","last_undegraded":"2024-09-16T08:59:31.204380+0000","last_fullsized":"2024-09-16T08:59:31.204380+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:01:13.616063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203350+0000","last_change":"2024-09-16T08:58:59.259671+0000","last_active":"2024-09-16T08:59:31.203350+0000","last_peered":"2024-09-16T08:59:31.203350+0000","last_clean":"2024-09-16T08:59:31.203350+0000","last_became_active":"2024-09-16T08:58:59.259401+0000","last_became_peered":"2024-09-16T08:58:59.259401+0000","last_unstale":"2024-09-16T08:59:31.203350+0000","last_undegraded":"2024-09-16T08:59:31.203350+0000","last_fullsized":"2024-09-16T08:59:31.203350+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:27:04.456124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:40.088326+0000","last_change":"2024-09-16T08:59:01.433442+0000","last_active":"2024-09-16T08:59:40.088326+0000","last_peered":"2024-09-16T08:59:40.088326+0000","last_clean":"2024-09-16T08:59:40.088326+0000","last_became_active":"2024-09-16T08:58:58.248425+0000","last_became_peered":"2024-09-16T08:58:58.248425+0000","last_unstale":"2024-09-16T08:59:40.088326+0000","last_undegraded":"2024-09-16T08:59:40.088326+0000","last_fullsized":"2024-09-16T08:59:40.088326+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_clean_scrub_stamp":"2024-09-16T08:58:57.230240+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:13:32.978470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069780000000000005,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":32,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.776001+0000","last_change":"2024-09-16T08:59:03.281201+0000","last_active":"2024-09-16T08:59:30.776001+0000","last_peered":"2024-09-16T08:59:30.776001+0000","last_clean":"2024-09-16T08:59:30.776001+0000","last_became_active":"2024-09-16T08:59:03.280751+0000","last_became_peered":"2024-09-16T08:59:03.280751+0000","last_unstale":"2024-09-16T08:59:30.776001+0000","last_undegraded":"2024-09-16T08:59:30.776001+0000","last_fullsized":"2024-09-16T08:59:30.776001+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:02:08.970051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439478+0000","last_change":"2024-09-16T08:59:05.290351+0000","last_active":"2024-09-16T08:59:30.439478+0000","last_peered":"2024-09-16T08:59:30.439478+0000","last_clean":"2024-09-16T08:59:30.439478+0000","last_became_active":"2024-09-16T08:59:05.289470+0000","last_became_peered":"2024-09-16T08:59:05.289470+0000","last_unstale":"2024-09-16T08:59:30.439478+0000","last_undegraded":"2024-09-16T08:59:30.439478+0000","last_fullsized":"2024-09-16T08:59:30.439478+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:48.796868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"61'16","reported_seq":57,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318308+0000","last_change":"2024-09-16T08:59:01.435018+0000","last_active":"2024-09-16T08:59:34.318308+0000","last_peered":"2024-09-16T08:59:34.318308+0000","last_clean":"2024-09-16T08:59:34.318308+0000","last_became_active":"2024-09-16T08:59:01.434721+0000","last_became_peered":"2024-09-16T08:59:01.434721+0000","last_unstale":"2024-09-16T08:59:34.318308+0000","last_undegraded":"2024-09-16T08:59:34.318308+0000","last_fullsized":"2024-09-16T08:59:34.318308+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:42.846463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440652+0000","last_change":"2024-09-16T08:58:59.263361+0000","last_active":"2024-09-16T08:59:30.440652+0000","last_peered":"2024-09-16T08:59:30.440652+0000","last_clean":"2024-09-16T08:59:30.440652+0000","last_became_active":"2024-09-16T08:58:59.263074+0000","last_became_peered":"2024-09-16T08:58:59.263074+0000","last_unstale":"2024-09-16T08:59:30.440652+0000","last_undegraded":"2024-09-16T08:59:30.440652+0000","last_fullsized":"2024-09-16T08:59:30.440652+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:01.696844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"63'79","reported_seq":67,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:32.763949+0000","last_change":"2024-09-16T08:58:49.325063+0000","last_active":"2024-09-16T08:59:32.763949+0000","last_peered":"2024-09-16T08:59:32.763949+0000","last_clean":"2024-09-16T08:59:32.763949+0000","last_became_active":"2024-09-16T08:58:49.319994+0000","last_became_peered":"2024-09-16T08:58:49.319994+0000","last_unstale":"2024-09-16T08:59:32.763949+0000","last_undegraded":"2024-09-16T08:59:32.763949+0000","last_fullsized":"2024-09-16T08:59:32.763949+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:56:27.881800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:56:27.881800+0000","last_clean_scrub_stamp":"2024-09-16T08:56:27.881800+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:11:17.852949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440659+0000","last_change":"2024-09-16T08:59:03.285822+0000","last_active":"2024-09-16T08:59:30.440659+0000","last_peered":"2024-09-16T08:59:30.440659+0000","last_clean":"2024-09-16T08:59:30.440659+0000","last_became_active":"2024-09-16T08:59:03.284965+0000","last_became_peered":"2024-09-16T08:59:03.284965+0000","last_unstale":"2024-09-16T08:59:30.440659+0000","last_undegraded":"2024-09-16T08:59:30.440659+0000","last_fullsized":"2024-09-16T08:59:30.440659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:49:30.467506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775851+0000","last_change":"2024-09-16T08:59:05.291694+0000","last_active":"2024-09-16T08:59:30.775851+0000","last_peered":"2024-09-16T08:59:30.775851+0000","last_clean":"2024-09-16T08:59:30.775851+0000","last_became_active":"2024-09-16T08:59:05.291287+0000","last_became_peered":"2024-09-16T08:59:05.291287+0000","last_unstale":"2024-09-16T08:59:30.775851+0000","last_undegraded":"2024-09-16T08:59:30.775851+0000","last_fullsized":"2024-09-16T08:59:30.775851+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:50:52.992278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"61'11","reported_seq":45,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204331+0000","last_change":"2024-09-16T08:59:01.266728+0000","last_active":"2024-09-16T08:59:31.204331+0000","last_peered":"2024-09-16T08:59:31.204331+0000","last_clean":"2024-09-16T08:59:31.204331+0000","last_became_active":"2024-09-16T08:59:01.266389+0000","last_became_peered":"2024-09-16T08:59:01.266389+0000","last_unstale":"2024-09-16T08:59:31.204331+0000","last_undegraded":"2024-09-16T08:59:31.204331+0000","last_fullsized":"2024-09-16T08:59:31.204331+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:01:03.005036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203599+0000","last_change":"2024-09-16T08:58:59.260106+0000","last_active":"2024-09-16T08:59:31.203599+0000","last_peered":"2024-09-16T08:59:31.203599+0000","last_clean":"2024-09-16T08:59:31.203599+0000","last_became_active":"2024-09-16T08:58:59.259815+0000","last_became_peered":"2024-09-16T08:58:59.259815+0000","last_unstale":"2024-09-16T08:59:31.203599+0000","last_undegraded":"2024-09-16T08:59:31.203599+0000","last_fullsized":"2024-09-16T08:59:31.203599+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:06.550778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775912+0000","last_change":"2024-09-16T08:59:03.284575+0000","last_active":"2024-09-16T08:59:30.775912+0000","last_peered":"2024-09-16T08:59:30.775912+0000","last_clean":"2024-09-16T08:59:30.775912+0000","last_became_active":"2024-09-16T08:59:03.284214+0000","last_became_peered":"2024-09-16T08:59:03.284214+0000","last_unstale":"2024-09-16T08:59:30.775912+0000","last_undegraded":"2024-09-16T08:59:30.775912+0000","last_fullsized":"2024-09-16T08:59:30.775912+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:37.032833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440134+0000","last_change":"2024-09-16T08:59:05.297367+0000","last_active":"2024-09-16T08:59:30.440134+0000","last_peered":"2024-09-16T08:59:30.440134+0000","last_clean":"2024-09-16T08:59:30.440134+0000","last_became_active":"2024-09-16T08:59:05.297010+0000","last_became_peered":"2024-09-16T08:59:05.297010+0000","last_unstale":"2024-09-16T08:59:30.440134+0000","last_undegraded":"2024-09-16T08:59:30.440134+0000","last_fullsized":"2024-09-16T08:59:30.440134+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:55.243088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"61'17","reported_seq":54,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204265+0000","last_change":"2024-09-16T08:59:01.265603+0000","last_active":"2024-09-16T08:59:31.204265+0000","last_peered":"2024-09-16T08:59:31.204265+0000","last_clean":"2024-09-16T08:59:31.204265+0000","last_became_active":"2024-09-16T08:59:01.265279+0000","last_became_peered":"2024-09-16T08:59:01.265279+0000","last_unstale":"2024-09-16T08:59:31.204265+0000","last_undegraded":"2024-09-16T08:59:31.204265+0000","last_fullsized":"2024-09-16T08:59:31.204265+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:02.792134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203350+0000","last_change":"2024-09-16T08:58:59.259712+0000","last_active":"2024-09-16T08:59:31.203350+0000","last_peered":"2024-09-16T08:59:31.203350+0000","last_clean":"2024-09-16T08:59:31.203350+0000","last_became_active":"2024-09-16T08:58:59.259390+0000","last_became_peered":"2024-09-16T08:58:59.259390+0000","last_unstale":"2024-09-16T08:59:31.203350+0000","last_undegraded":"2024-09-16T08:59:31.203350+0000","last_fullsized":"2024-09-16T08:59:31.203350+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:34.944796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439575+0000","last_change":"2024-09-16T08:59:03.279579+0000","last_active":"2024-09-16T08:59:30.439575+0000","last_peered":"2024-09-16T08:59:30.439575+0000","last_clean":"2024-09-16T08:59:30.439575+0000","last_became_active":"2024-09-16T08:59:03.279283+0000","last_became_peered":"2024-09-16T08:59:03.279283+0000","last_unstale":"2024-09-16T08:59:30.439575+0000","last_undegraded":"2024-09-16T08:59:30.439575+0000","last_fullsized":"2024-09-16T08:59:30.439575+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:56.034243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781818+0000","last_change":"2024-09-16T08:59:05.292792+0000","last_active":"2024-09-16T08:59:30.781818+0000","last_peered":"2024-09-16T08:59:30.781818+0000","last_clean":"2024-09-16T08:59:30.781818+0000","last_became_active":"2024-09-16T08:59:05.292378+0000","last_became_peered":"2024-09-16T08:59:05.292378+0000","last_unstale":"2024-09-16T08:59:30.781818+0000","last_undegraded":"2024-09-16T08:59:30.781818+0000","last_fullsized":"2024-09-16T08:59:30.781818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:09:38.969129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"61'10","reported_seq":41,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204026+0000","last_change":"2024-09-16T08:59:01.264761+0000","last_active":"2024-09-16T08:59:31.204026+0000","last_peered":"2024-09-16T08:59:31.204026+0000","last_clean":"2024-09-16T08:59:31.204026+0000","last_became_active":"2024-09-16T08:59:01.264348+0000","last_became_peered":"2024-09-16T08:59:01.264348+0000","last_unstale":"2024-09-16T08:59:31.204026+0000","last_undegraded":"2024-09-16T08:59:31.204026+0000","last_fullsized":"2024-09-16T08:59:31.204026+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:33.406515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441000+0000","last_change":"2024-09-16T08:58:59.256842+0000","last_active":"2024-09-16T08:59:30.441000+0000","last_peered":"2024-09-16T08:59:30.441000+0000","last_clean":"2024-09-16T08:59:30.441000+0000","last_became_active":"2024-09-16T08:58:59.256544+0000","last_became_peered":"2024-09-16T08:58:59.256544+0000","last_unstale":"2024-09-16T08:59:30.441000+0000","last_undegraded":"2024-09-16T08:59:30.441000+0000","last_fullsized":"2024-09-16T08:59:30.441000+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:42.174926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318247+0000","last_change":"2024-09-16T08:59:03.280982+0000","last_active":"2024-09-16T08:59:34.318247+0000","last_peered":"2024-09-16T08:59:34.318247+0000","last_clean":"2024-09-16T08:59:34.318247+0000","last_became_active":"2024-09-16T08:59:03.280167+0000","last_became_peered":"2024-09-16T08:59:03.280167+0000","last_unstale":"2024-09-16T08:59:34.318247+0000","last_undegraded":"2024-09-16T08:59:34.318247+0000","last_fullsized":"2024-09-16T08:59:34.318247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:05.152692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204197+0000","last_change":"2024-09-16T08:59:05.289377+0000","last_active":"2024-09-16T08:59:31.204197+0000","last_peered":"2024-09-16T08:59:31.204197+0000","last_clean":"2024-09-16T08:59:31.204197+0000","last_became_active":"2024-09-16T08:59:05.289089+0000","last_became_peered":"2024-09-16T08:59:05.289089+0000","last_unstale":"2024-09-16T08:59:31.204197+0000","last_undegraded":"2024-09-16T08:59:31.204197+0000","last_fullsized":"2024-09-16T08:59:31.204197+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:54.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204510+0000","last_change":"2024-09-16T08:59:01.274085+0000","last_active":"2024-09-16T08:59:31.204510+0000","last_peered":"2024-09-16T08:59:31.204510+0000","last_clean":"2024-09-16T08:59:31.204510+0000","last_became_active":"2024-09-16T08:59:01.273619+0000","last_became_peered":"2024-09-16T08:59:01.273619+0000","last_unstale":"2024-09-16T08:59:31.204510+0000","last_undegraded":"2024-09-16T08:59:31.204510+0000","last_fullsized":"2024-09-16T08:59:31.204510+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:20:40.339908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775910+0000","last_change":"2024-09-16T08:58:59.263139+0000","last_active":"2024-09-16T08:59:30.775910+0000","last_peered":"2024-09-16T08:59:30.775910+0000","last_clean":"2024-09-16T08:59:30.775910+0000","last_became_active":"2024-09-16T08:58:59.262812+0000","last_became_peered":"2024-09-16T08:58:59.262812+0000","last_unstale":"2024-09-16T08:59:30.775910+0000","last_undegraded":"2024-09-16T08:59:30.775910+0000","last_fullsized":"2024-09-16T08:59:30.775910+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:03.695315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781318+0000","last_change":"2024-09-16T08:59:03.275837+0000","last_active":"2024-09-16T08:59:30.781318+0000","last_peered":"2024-09-16T08:59:30.781318+0000","last_clean":"2024-09-16T08:59:30.781318+0000","last_became_active":"2024-09-16T08:59:03.275550+0000","last_became_peered":"2024-09-16T08:59:03.275550+0000","last_unstale":"2024-09-16T08:59:30.781318+0000","last_undegraded":"2024-09-16T08:59:30.781318+0000","last_fullsized":"2024-09-16T08:59:30.781318+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:43.616278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203251+0000","last_change":"2024-09-16T08:59:05.290834+0000","last_active":"2024-09-16T08:59:31.203251+0000","last_peered":"2024-09-16T08:59:31.203251+0000","last_clean":"2024-09-16T08:59:31.203251+0000","last_became_active":"2024-09-16T08:59:05.290449+0000","last_became_peered":"2024-09-16T08:59:05.290449+0000","last_unstale":"2024-09-16T08:59:31.203251+0000","last_undegraded":"2024-09-16T08:59:31.203251+0000","last_fullsized":"2024-09-16T08:59:31.203251+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:48:51.370372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"61'19","reported_seq":57,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203152+0000","last_change":"2024-09-16T08:59:01.434403+0000","last_active":"2024-09-16T08:59:31.203152+0000","last_peered":"2024-09-16T08:59:31.203152+0000","last_clean":"2024-09-16T08:59:31.203152+0000","last_became_active":"2024-09-16T08:59:01.433965+0000","last_became_peered":"2024-09-16T08:59:01.433965+0000","last_unstale":"2024-09-16T08:59:31.203152+0000","last_undegraded":"2024-09-16T08:59:31.203152+0000","last_fullsized":"2024-09-16T08:59:31.203152+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:30.542133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440989+0000","last_change":"2024-09-16T08:58:59.266874+0000","last_active":"2024-09-16T08:59:30.440989+0000","last_peered":"2024-09-16T08:59:30.440989+0000","last_clean":"2024-09-16T08:59:30.440989+0000","last_became_active":"2024-09-16T08:58:59.266555+0000","last_became_peered":"2024-09-16T08:58:59.266555+0000","last_unstale":"2024-09-16T08:59:30.440989+0000","last_undegraded":"2024-09-16T08:59:30.440989+0000","last_fullsized":"2024-09-16T08:59:30.440989+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:15.315810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781394+0000","last_change":"2024-09-16T08:59:03.281576+0000","last_active":"2024-09-16T08:59:30.781394+0000","last_peered":"2024-09-16T08:59:30.781394+0000","last_clean":"2024-09-16T08:59:30.781394+0000","last_became_active":"2024-09-16T08:59:03.280148+0000","last_became_peered":"2024-09-16T08:59:03.280148+0000","last_unstale":"2024-09-16T08:59:30.781394+0000","last_undegraded":"2024-09-16T08:59:30.781394+0000","last_fullsized":"2024-09-16T08:59:30.781394+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:35.787588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440034+0000","last_change":"2024-09-16T08:59:05.289415+0000","last_active":"2024-09-16T08:59:30.440034+0000","last_peered":"2024-09-16T08:59:30.440034+0000","last_clean":"2024-09-16T08:59:30.440034+0000","last_became_active":"2024-09-16T08:59:05.288932+0000","last_became_peered":"2024-09-16T08:59:05.288932+0000","last_unstale":"2024-09-16T08:59:30.440034+0000","last_undegraded":"2024-09-16T08:59:30.440034+0000","last_fullsized":"2024-09-16T08:59:30.440034+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:17:27.315454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"61'12","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203246+0000","last_change":"2024-09-16T08:59:01.266118+0000","last_active":"2024-09-16T08:59:31.203246+0000","last_peered":"2024-09-16T08:59:31.203246+0000","last_clean":"2024-09-16T08:59:31.203246+0000","last_became_active":"2024-09-16T08:59:01.264492+0000","last_became_peered":"2024-09-16T08:59:01.264492+0000","last_unstale":"2024-09-16T08:59:31.203246+0000","last_undegraded":"2024-09-16T08:59:31.203246+0000","last_fullsized":"2024-09-16T08:59:31.203246+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:46:25.022599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441869+0000","last_change":"2024-09-16T08:58:59.266794+0000","last_active":"2024-09-16T08:59:30.441869+0000","last_peered":"2024-09-16T08:59:30.441869+0000","last_clean":"2024-09-16T08:59:30.441869+0000","last_became_active":"2024-09-16T08:58:59.266467+0000","last_became_peered":"2024-09-16T08:58:59.266467+0000","last_unstale":"2024-09-16T08:59:30.441869+0000","last_undegraded":"2024-09-16T08:59:30.441869+0000","last_fullsized":"2024-09-16T08:59:30.441869+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:13.704059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781285+0000","last_change":"2024-09-16T08:59:03.281208+0000","last_active":"2024-09-16T08:59:30.781285+0000","last_peered":"2024-09-16T08:59:30.781285+0000","last_clean":"2024-09-16T08:59:30.781285+0000","last_became_active":"2024-09-16T08:59:03.280931+0000","last_became_peered":"2024-09-16T08:59:03.280931+0000","last_unstale":"2024-09-16T08:59:30.781285+0000","last_undegraded":"2024-09-16T08:59:30.781285+0000","last_fullsized":"2024-09-16T08:59:30.781285+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:08:45.251981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440692+0000","last_change":"2024-09-16T08:59:05.290572+0000","last_active":"2024-09-16T08:59:30.440692+0000","last_peered":"2024-09-16T08:59:30.440692+0000","last_clean":"2024-09-16T08:59:30.440692+0000","last_became_active":"2024-09-16T08:59:05.290273+0000","last_became_peered":"2024-09-16T08:59:05.290273+0000","last_unstale":"2024-09-16T08:59:30.440692+0000","last_undegraded":"2024-09-16T08:59:30.440692+0000","last_fullsized":"2024-09-16T08:59:30.440692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:02:52.134683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"61'15","reported_seq":51,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775300+0000","last_change":"2024-09-16T08:59:01.434570+0000","last_active":"2024-09-16T08:59:30.775300+0000","last_peered":"2024-09-16T08:59:30.775300+0000","last_clean":"2024-09-16T08:59:30.775300+0000","last_became_active":"2024-09-16T08:59:01.434251+0000","last_became_peered":"2024-09-16T08:59:01.434251+0000","last_unstale":"2024-09-16T08:59:30.775300+0000","last_undegraded":"2024-09-16T08:59:30.775300+0000","last_fullsized":"2024-09-16T08:59:30.775300+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:17:30.165936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"54'2","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441572+0000","last_change":"2024-09-16T08:58:59.266786+0000","last_active":"2024-09-16T08:59:30.441572+0000","last_peered":"2024-09-16T08:59:30.441572+0000","last_clean":"2024-09-16T08:59:30.441572+0000","last_became_active":"2024-09-16T08:58:59.266445+0000","last_became_peered":"2024-09-16T08:58:59.266445+0000","last_unstale":"2024-09-16T08:59:30.441572+0000","last_undegraded":"2024-09-16T08:59:30.441572+0000","last_fullsized":"2024-09-16T08:59:30.441572+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:53:46.828836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318212+0000","last_change":"2024-09-16T08:59:03.283746+0000","last_active":"2024-09-16T08:59:34.318212+0000","last_peered":"2024-09-16T08:59:34.318212+0000","last_clean":"2024-09-16T08:59:34.318212+0000","last_became_active":"2024-09-16T08:59:03.283346+0000","last_became_peered":"2024-09-16T08:59:03.283346+0000","last_unstale":"2024-09-16T08:59:34.318212+0000","last_undegraded":"2024-09-16T08:59:34.318212+0000","last_fullsized":"2024-09-16T08:59:34.318212+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:39.352949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775405+0000","last_change":"2024-09-16T08:59:05.292279+0000","last_active":"2024-09-16T08:59:30.775405+0000","last_peered":"2024-09-16T08:59:30.775405+0000","last_clean":"2024-09-16T08:59:30.775405+0000","last_became_active":"2024-09-16T08:59:05.291768+0000","last_became_peered":"2024-09-16T08:59:05.291768+0000","last_unstale":"2024-09-16T08:59:30.775405+0000","last_undegraded":"2024-09-16T08:59:30.775405+0000","last_fullsized":"2024-09-16T08:59:30.775405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:26.003718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.202800+0000","last_change":"2024-09-16T08:58:59.259063+0000","last_active":"2024-09-16T08:59:31.202800+0000","last_peered":"2024-09-16T08:59:31.202800+0000","last_clean":"2024-09-16T08:59:31.202800+0000","last_became_active":"2024-09-16T08:58:59.258752+0000","last_became_peered":"2024-09-16T08:58:59.258752+0000","last_unstale":"2024-09-16T08:59:31.202800+0000","last_undegraded":"2024-09-16T08:59:31.202800+0000","last_fullsized":"2024-09-16T08:59:31.202800+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:33:46.685367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"61'6","reported_seq":35,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441432+0000","last_change":"2024-09-16T08:59:01.434607+0000","last_active":"2024-09-16T08:59:30.441432+0000","last_peered":"2024-09-16T08:59:30.441432+0000","last_clean":"2024-09-16T08:59:30.441432+0000","last_became_active":"2024-09-16T08:59:01.434292+0000","last_became_peered":"2024-09-16T08:59:01.434292+0000","last_unstale":"2024-09-16T08:59:30.441432+0000","last_undegraded":"2024-09-16T08:59:30.441432+0000","last_fullsized":"2024-09-16T08:59:30.441432+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:09.549373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775780+0000","last_change":"2024-09-16T08:59:03.281523+0000","last_active":"2024-09-16T08:59:30.775780+0000","last_peered":"2024-09-16T08:59:30.775780+0000","last_clean":"2024-09-16T08:59:30.775780+0000","last_became_active":"2024-09-16T08:59:03.280220+0000","last_became_peered":"2024-09-16T08:59:03.280220+0000","last_unstale":"2024-09-16T08:59:30.775780+0000","last_undegraded":"2024-09-16T08:59:30.775780+0000","last_fullsized":"2024-09-16T08:59:30.775780+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:20:36.292718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440663+0000","last_change":"2024-09-16T08:59:05.292037+0000","last_active":"2024-09-16T08:59:30.440663+0000","last_peered":"2024-09-16T08:59:30.440663+0000","last_clean":"2024-09-16T08:59:30.440663+0000","last_became_active":"2024-09-16T08:59:05.291641+0000","last_became_peered":"2024-09-16T08:59:05.291641+0000","last_unstale":"2024-09-16T08:59:30.440663+0000","last_undegraded":"2024-09-16T08:59:30.440663+0000","last_fullsized":"2024-09-16T08:59:30.440663+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:00.620413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204540+0000","last_change":"2024-09-16T08:59:01.433875+0000","last_active":"2024-09-16T08:59:31.204540+0000","last_peered":"2024-09-16T08:59:31.204540+0000","last_clean":"2024-09-16T08:59:31.204540+0000","last_became_active":"2024-09-16T08:59:01.433570+0000","last_became_peered":"2024-09-16T08:59:01.433570+0000","last_unstale":"2024-09-16T08:59:31.204540+0000","last_undegraded":"2024-09-16T08:59:31.204540+0000","last_fullsized":"2024-09-16T08:59:31.204540+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:01:07.210788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440449+0000","last_change":"2024-09-16T08:58:59.265895+0000","last_active":"2024-09-16T08:59:30.440449+0000","last_peered":"2024-09-16T08:59:30.440449+0000","last_clean":"2024-09-16T08:59:30.440449+0000","last_became_active":"2024-09-16T08:58:59.265588+0000","last_became_peered":"2024-09-16T08:58:59.265588+0000","last_unstale":"2024-09-16T08:59:30.440449+0000","last_undegraded":"2024-09-16T08:59:30.440449+0000","last_fullsized":"2024-09-16T08:59:30.440449+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:57:40.324659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441286+0000","last_change":"2024-09-16T08:59:03.282785+0000","last_active":"2024-09-16T08:59:30.441286+0000","last_peered":"2024-09-16T08:59:30.441286+0000","last_clean":"2024-09-16T08:59:30.441286+0000","last_became_active":"2024-09-16T08:59:03.282524+0000","last_became_peered":"2024-09-16T08:59:03.282524+0000","last_unstale":"2024-09-16T08:59:30.441286+0000","last_undegraded":"2024-09-16T08:59:30.441286+0000","last_fullsized":"2024-09-16T08:59:30.441286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:31.268238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781599+0000","last_change":"2024-09-16T08:59:05.291726+0000","last_active":"2024-09-16T08:59:30.781599+0000","last_peered":"2024-09-16T08:59:30.781599+0000","last_clean":"2024-09-16T08:59:30.781599+0000","last_became_active":"2024-09-16T08:59:05.291390+0000","last_became_peered":"2024-09-16T08:59:05.291390+0000","last_unstale":"2024-09-16T08:59:30.781599+0000","last_undegraded":"2024-09-16T08:59:30.781599+0000","last_fullsized":"2024-09-16T08:59:30.781599+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:49:50.012693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775829+0000","last_change":"2024-09-16T08:59:01.433869+0000","last_active":"2024-09-16T08:59:30.775829+0000","last_peered":"2024-09-16T08:59:30.775829+0000","last_clean":"2024-09-16T08:59:30.775829+0000","last_became_active":"2024-09-16T08:59:01.433531+0000","last_became_peered":"2024-09-16T08:59:01.433531+0000","last_unstale":"2024-09-16T08:59:30.775829+0000","last_undegraded":"2024-09-16T08:59:30.775829+0000","last_fullsized":"2024-09-16T08:59:30.775829+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:32:51.504027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203901+0000","last_change":"2024-09-16T08:58:59.260819+0000","last_active":"2024-09-16T08:59:31.203901+0000","last_peered":"2024-09-16T08:59:31.203901+0000","last_clean":"2024-09-16T08:59:31.203901+0000","last_became_active":"2024-09-16T08:58:59.260293+0000","last_became_peered":"2024-09-16T08:58:59.260293+0000","last_unstale":"2024-09-16T08:59:31.203901+0000","last_undegraded":"2024-09-16T08:59:31.203901+0000","last_fullsized":"2024-09-16T08:59:31.203901+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:53:11.123625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318372+0000","last_change":"2024-09-16T08:59:03.281887+0000","last_active":"2024-09-16T08:59:34.318372+0000","last_peered":"2024-09-16T08:59:34.318372+0000","last_clean":"2024-09-16T08:59:34.318372+0000","last_became_active":"2024-09-16T08:59:03.281598+0000","last_became_peered":"2024-09-16T08:59:03.281598+0000","last_unstale":"2024-09-16T08:59:34.318372+0000","last_undegraded":"2024-09-16T08:59:34.318372+0000","last_fullsized":"2024-09-16T08:59:34.318372+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:10:52.288798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204400+0000","last_change":"2024-09-16T08:59:05.292678+0000","last_active":"2024-09-16T08:59:31.204400+0000","last_peered":"2024-09-16T08:59:31.204400+0000","last_clean":"2024-09-16T08:59:31.204400+0000","last_became_active":"2024-09-16T08:59:05.292409+0000","last_became_peered":"2024-09-16T08:59:05.292409+0000","last_unstale":"2024-09-16T08:59:31.204400+0000","last_undegraded":"2024-09-16T08:59:31.204400+0000","last_fullsized":"2024-09-16T08:59:31.204400+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:22.566068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"61'10","reported_seq":41,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441392+0000","last_change":"2024-09-16T08:59:01.433702+0000","last_active":"2024-09-16T08:59:30.441392+0000","last_peered":"2024-09-16T08:59:30.441392+0000","last_clean":"2024-09-16T08:59:30.441392+0000","last_became_active":"2024-09-16T08:59:01.433397+0000","last_became_peered":"2024-09-16T08:59:01.433397+0000","last_unstale":"2024-09-16T08:59:30.441392+0000","last_undegraded":"2024-09-16T08:59:30.441392+0000","last_fullsized":"2024-09-16T08:59:30.441392+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:57.100505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440863+0000","last_change":"2024-09-16T08:58:59.264578+0000","last_active":"2024-09-16T08:59:30.440863+0000","last_peered":"2024-09-16T08:59:30.440863+0000","last_clean":"2024-09-16T08:59:30.440863+0000","last_became_active":"2024-09-16T08:58:59.264258+0000","last_became_peered":"2024-09-16T08:58:59.264258+0000","last_unstale":"2024-09-16T08:59:30.440863+0000","last_undegraded":"2024-09-16T08:59:30.440863+0000","last_fullsized":"2024-09-16T08:59:30.440863+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:25.824717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318207+0000","last_change":"2024-09-16T08:59:03.281319+0000","last_active":"2024-09-16T08:59:34.318207+0000","last_peered":"2024-09-16T08:59:34.318207+0000","last_clean":"2024-09-16T08:59:34.318207+0000","last_became_active":"2024-09-16T08:59:03.281033+0000","last_became_peered":"2024-09-16T08:59:03.281033+0000","last_unstale":"2024-09-16T08:59:34.318207+0000","last_undegraded":"2024-09-16T08:59:34.318207+0000","last_fullsized":"2024-09-16T08:59:34.318207+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:17:32.662151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204002+0000","last_change":"2024-09-16T08:59:05.291676+0000","last_active":"2024-09-16T08:59:31.204002+0000","last_peered":"2024-09-16T08:59:31.204002+0000","last_clean":"2024-09-16T08:59:31.204002+0000","last_became_active":"2024-09-16T08:59:05.291390+0000","last_became_peered":"2024-09-16T08:59:05.291390+0000","last_unstale":"2024-09-16T08:59:31.204002+0000","last_undegraded":"2024-09-16T08:59:31.204002+0000","last_fullsized":"2024-09-16T08:59:31.204002+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:09:09.160434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"61'11","reported_seq":45,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203914+0000","last_change":"2024-09-16T08:59:01.268104+0000","last_active":"2024-09-16T08:59:31.203914+0000","last_peered":"2024-09-16T08:59:31.203914+0000","last_clean":"2024-09-16T08:59:31.203914+0000","last_became_active":"2024-09-16T08:59:01.267720+0000","last_became_peered":"2024-09-16T08:59:01.267720+0000","last_unstale":"2024-09-16T08:59:31.203914+0000","last_undegraded":"2024-09-16T08:59:31.203914+0000","last_fullsized":"2024-09-16T08:59:31.203914+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:41.383063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204204+0000","last_change":"2024-09-16T08:58:59.266512+0000","last_active":"2024-09-16T08:59:31.204204+0000","last_peered":"2024-09-16T08:59:31.204204+0000","last_clean":"2024-09-16T08:59:31.204204+0000","last_became_active":"2024-09-16T08:58:59.266218+0000","last_became_peered":"2024-09-16T08:58:59.266218+0000","last_unstale":"2024-09-16T08:59:31.204204+0000","last_undegraded":"2024-09-16T08:59:31.204204+0000","last_fullsized":"2024-09-16T08:59:31.204204+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:45.360169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439294+0000","last_change":"2024-09-16T08:59:03.285503+0000","last_active":"2024-09-16T08:59:30.439294+0000","last_peered":"2024-09-16T08:59:30.439294+0000","last_clean":"2024-09-16T08:59:30.439294+0000","last_became_active":"2024-09-16T08:59:03.285244+0000","last_became_peered":"2024-09-16T08:59:03.285244+0000","last_unstale":"2024-09-16T08:59:30.439294+0000","last_undegraded":"2024-09-16T08:59:30.439294+0000","last_fullsized":"2024-09-16T08:59:30.439294+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:55:04.012390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440024+0000","last_change":"2024-09-16T08:59:05.296956+0000","last_active":"2024-09-16T08:59:30.440024+0000","last_peered":"2024-09-16T08:59:30.440024+0000","last_clean":"2024-09-16T08:59:30.440024+0000","last_became_active":"2024-09-16T08:59:05.296628+0000","last_became_peered":"2024-09-16T08:59:05.296628+0000","last_unstale":"2024-09-16T08:59:30.440024+0000","last_undegraded":"2024-09-16T08:59:30.440024+0000","last_fullsized":"2024-09-16T08:59:30.440024+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:18:54.119159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439622+0000","last_change":"2024-09-16T08:59:01.267926+0000","last_active":"2024-09-16T08:59:30.439622+0000","last_peered":"2024-09-16T08:59:30.439622+0000","last_clean":"2024-09-16T08:59:30.439622+0000","last_became_active":"2024-09-16T08:59:01.267622+0000","last_became_peered":"2024-09-16T08:59:01.267622+0000","last_unstale":"2024-09-16T08:59:30.439622+0000","last_undegraded":"2024-09-16T08:59:30.439622+0000","last_fullsized":"2024-09-16T08:59:30.439622+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:34:53.973215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440848+0000","last_change":"2024-09-16T08:58:59.267394+0000","last_active":"2024-09-16T08:59:30.440848+0000","last_peered":"2024-09-16T08:59:30.440848+0000","last_clean":"2024-09-16T08:59:30.440848+0000","last_became_active":"2024-09-16T08:58:59.267053+0000","last_became_peered":"2024-09-16T08:58:59.267053+0000","last_unstale":"2024-09-16T08:59:30.440848+0000","last_undegraded":"2024-09-16T08:59:30.440848+0000","last_fullsized":"2024-09-16T08:59:30.440848+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:53.938611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440742+0000","last_change":"2024-09-16T08:59:03.280783+0000","last_active":"2024-09-16T08:59:30.440742+0000","last_peered":"2024-09-16T08:59:30.440742+0000","last_clean":"2024-09-16T08:59:30.440742+0000","last_became_active":"2024-09-16T08:59:03.280492+0000","last_became_peered":"2024-09-16T08:59:03.280492+0000","last_unstale":"2024-09-16T08:59:30.440742+0000","last_undegraded":"2024-09-16T08:59:30.440742+0000","last_fullsized":"2024-09-16T08:59:30.440742+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:48.132611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203029+0000","last_change":"2024-09-16T08:59:05.287326+0000","last_active":"2024-09-16T08:59:31.203029+0000","last_peered":"2024-09-16T08:59:31.203029+0000","last_clean":"2024-09-16T08:59:31.203029+0000","last_became_active":"2024-09-16T08:59:05.287023+0000","last_became_peered":"2024-09-16T08:59:05.287023+0000","last_unstale":"2024-09-16T08:59:31.203029+0000","last_undegraded":"2024-09-16T08:59:31.203029+0000","last_fullsized":"2024-09-16T08:59:31.203029+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:48:38.507456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"61'11","reported_seq":45,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440156+0000","last_change":"2024-09-16T08:59:01.433487+0000","last_active":"2024-09-16T08:59:30.440156+0000","last_peered":"2024-09-16T08:59:30.440156+0000","last_clean":"2024-09-16T08:59:30.440156+0000","last_became_active":"2024-09-16T08:59:01.433074+0000","last_became_peered":"2024-09-16T08:59:01.433074+0000","last_unstale":"2024-09-16T08:59:30.440156+0000","last_undegraded":"2024-09-16T08:59:30.440156+0000","last_fullsized":"2024-09-16T08:59:30.440156+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:05:19.437647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775821+0000","last_change":"2024-09-16T08:58:59.257127+0000","last_active":"2024-09-16T08:59:30.775821+0000","last_peered":"2024-09-16T08:59:30.775821+0000","last_clean":"2024-09-16T08:59:30.775821+0000","last_became_active":"2024-09-16T08:58:59.256814+0000","last_became_peered":"2024-09-16T08:58:59.256814+0000","last_unstale":"2024-09-16T08:59:30.775821+0000","last_undegraded":"2024-09-16T08:59:30.775821+0000","last_fullsized":"2024-09-16T08:59:30.775821+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:46.796699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439842+0000","last_change":"2024-09-16T08:59:03.284410+0000","last_active":"2024-09-16T08:59:30.439842+0000","last_peered":"2024-09-16T08:59:30.439842+0000","last_clean":"2024-09-16T08:59:30.439842+0000","last_became_active":"2024-09-16T08:59:03.284139+0000","last_became_peered":"2024-09-16T08:59:03.284139+0000","last_unstale":"2024-09-16T08:59:30.439842+0000","last_undegraded":"2024-09-16T08:59:30.439842+0000","last_fullsized":"2024-09-16T08:59:30.439842+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:41:05.484174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440910+0000","last_change":"2024-09-16T08:59:05.294468+0000","last_active":"2024-09-16T08:59:30.440910+0000","last_peered":"2024-09-16T08:59:30.440910+0000","last_clean":"2024-09-16T08:59:30.440910+0000","last_became_active":"2024-09-16T08:59:05.294148+0000","last_became_peered":"2024-09-16T08:59:05.294148+0000","last_unstale":"2024-09-16T08:59:30.440910+0000","last_undegraded":"2024-09-16T08:59:30.440910+0000","last_fullsized":"2024-09-16T08:59:30.440910+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:52:12.649060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"61'4","reported_seq":32,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440111+0000","last_change":"2024-09-16T08:59:01.270548+0000","last_active":"2024-09-16T08:59:30.440111+0000","last_peered":"2024-09-16T08:59:30.440111+0000","last_clean":"2024-09-16T08:59:30.440111+0000","last_became_active":"2024-09-16T08:59:01.270274+0000","last_became_peered":"2024-09-16T08:59:01.270274+0000","last_unstale":"2024-09-16T08:59:30.440111+0000","last_undegraded":"2024-09-16T08:59:30.440111+0000","last_fullsized":"2024-09-16T08:59:30.440111+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:18.014009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204734+0000","last_change":"2024-09-16T08:58:59.257328+0000","last_active":"2024-09-16T08:59:31.204734+0000","last_peered":"2024-09-16T08:59:31.204734+0000","last_clean":"2024-09-16T08:59:31.204734+0000","last_became_active":"2024-09-16T08:58:59.257061+0000","last_became_peered":"2024-09-16T08:58:59.257061+0000","last_unstale":"2024-09-16T08:59:31.204734+0000","last_undegraded":"2024-09-16T08:59:31.204734+0000","last_fullsized":"2024-09-16T08:59:31.204734+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:53:18.183635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.202825+0000","last_change":"2024-09-16T08:59:03.283069+0000","last_active":"2024-09-16T08:59:31.202825+0000","last_peered":"2024-09-16T08:59:31.202825+0000","last_clean":"2024-09-16T08:59:31.202825+0000","last_became_active":"2024-09-16T08:59:03.282791+0000","last_became_peered":"2024-09-16T08:59:03.282791+0000","last_unstale":"2024-09-16T08:59:31.202825+0000","last_undegraded":"2024-09-16T08:59:31.202825+0000","last_fullsized":"2024-09-16T08:59:31.202825+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:23:43.786012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"61'1","reported_seq":19,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440396+0000","last_change":"2024-09-16T08:59:05.290227+0000","last_active":"2024-09-16T08:59:30.440396+0000","last_peered":"2024-09-16T08:59:30.440396+0000","last_clean":"2024-09-16T08:59:30.440396+0000","last_became_active":"2024-09-16T08:59:05.289940+0000","last_became_peered":"2024-09-16T08:59:05.289940+0000","last_unstale":"2024-09-16T08:59:30.440396+0000","last_undegraded":"2024-09-16T08:59:30.440396+0000","last_fullsized":"2024-09-16T08:59:30.440396+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:18:13.275619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439478+0000","last_change":"2024-09-16T08:59:05.289802+0000","last_active":"2024-09-16T08:59:30.439478+0000","last_peered":"2024-09-16T08:59:30.439478+0000","last_clean":"2024-09-16T08:59:30.439478+0000","last_became_active":"2024-09-16T08:59:05.289414+0000","last_became_peered":"2024-09-16T08:59:05.289414+0000","last_unstale":"2024-09-16T08:59:30.439478+0000","last_undegraded":"2024-09-16T08:59:30.439478+0000","last_fullsized":"2024-09-16T08:59:30.439478+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:14.377503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"54'1","reported_seq":36,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440438+0000","last_change":"2024-09-16T08:58:59.255988+0000","last_active":"2024-09-16T08:59:30.440438+0000","last_peered":"2024-09-16T08:59:30.440438+0000","last_clean":"2024-09-16T08:59:30.440438+0000","last_became_active":"2024-09-16T08:58:59.255595+0000","last_became_peered":"2024-09-16T08:58:59.255595+0000","last_unstale":"2024-09-16T08:59:30.440438+0000","last_undegraded":"2024-09-16T08:59:30.440438+0000","last_fullsized":"2024-09-16T08:59:30.440438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:03.865357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"61'11","reported_seq":45,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203763+0000","last_change":"2024-09-16T08:59:01.271803+0000","last_active":"2024-09-16T08:59:31.203763+0000","last_peered":"2024-09-16T08:59:31.203763+0000","last_clean":"2024-09-16T08:59:31.203763+0000","last_became_active":"2024-09-16T08:59:01.271468+0000","last_became_peered":"2024-09-16T08:59:01.271468+0000","last_unstale":"2024-09-16T08:59:31.203763+0000","last_undegraded":"2024-09-16T08:59:31.203763+0000","last_fullsized":"2024-09-16T08:59:31.203763+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:31.826849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204157+0000","last_change":"2024-09-16T08:59:03.283781+0000","last_active":"2024-09-16T08:59:31.204157+0000","last_peered":"2024-09-16T08:59:31.204157+0000","last_clean":"2024-09-16T08:59:31.204157+0000","last_became_active":"2024-09-16T08:59:03.282976+0000","last_became_peered":"2024-09-16T08:59:03.282976+0000","last_unstale":"2024-09-16T08:59:31.204157+0000","last_undegraded":"2024-09-16T08:59:31.204157+0000","last_fullsized":"2024-09-16T08:59:31.204157+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:15.809446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":32,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397518,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32452,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712700,"statfs":{"total":95995035648,"available":95961804800,"internally_reserved":0,"allocated":1298432,"data_stored":877431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593747,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32428,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712724,"statfs":{"total":95995035648,"available":95961829376,"internally_reserved":0,"allocated":1273856,"data_stored":876333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789975,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":286324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":716800,"data_stored":287576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":720896,"data_stored":287292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":287119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":753664,"data_stored":289179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28300,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716852,"statfs":{"total":95995035648,"available":95966056448,"internally_reserved":0,"allocated":1302528,"data_stored":877563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T08:59:43.878 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T08:59:44.130 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:44.459 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:44 smithi111 ceph-mon[29803]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-16T08:59:44.459 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:44 smithi111 ceph-mon[29803]: from='client.24590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T08:59:44.459 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:44 smithi111 ceph-mon[40056]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-16T08:59:44.460 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:44 smithi111 ceph-mon[40056]: from='client.24590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T08:59:44.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:44 smithi114 ceph-mon[33315]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-16T08:59:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:44 smithi114 ceph-mon[33315]: from='client.24590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T08:59:44.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.683+0000 7f80f616f640 1 -- 172.21.15.111:0/1672882280 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80f0109110 msgr2=0x7f80f010f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:44.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.683+0000 7f80f616f640 1 --2- 172.21.15.111:0/1672882280 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80f0109110 0x7f80f010f910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009ca0 tx=0x7f80e802a8c0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.683+0000 7f80f616f640 1 -- 172.21.15.111:0/1672882280 shutdown_connections 2024-09-16T08:59:44.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.683+0000 7f80f616f640 1 --2- 172.21.15.111:0/1672882280 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80f0109110 0x7f80f010f910 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.683+0000 7f80f616f640 1 --2- 172.21.15.111:0/1672882280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f0108790 0x7f80f0108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.683+0000 7f80f616f640 1 --2- 172.21.15.111:0/1672882280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80f0102d80 0x7f80f0103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.683+0000 7f80f616f640 1 -- 172.21.15.111:0/1672882280 >> 172.21.15.111:0/1672882280 conn(0x7f80f00fe7d0 msgr2=0x7f80f0100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:44.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.683+0000 7f80f616f640 1 -- 172.21.15.111:0/1672882280 shutdown_connections 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.683+0000 7f80f616f640 1 -- 172.21.15.111:0/1672882280 wait complete. 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80f616f640 1 Processor -- start 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80f616f640 1 -- start start 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80f616f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f0102d80 0x7f80f01987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80f616f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80f0108790 0x7f80f0198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80f616f640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80f0109110 0x7f80f019e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80f616f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f80f01122a0 con 0x7f80f0102d80 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80f616f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f80f0112120 con 0x7f80f0108790 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80f616f640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f80f0112420 con 0x7f80f0109110 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80ef7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f0102d80 0x7f80f01987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80effff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80f0109110 0x7f80f019e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80ef7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f0102d80 0x7f80f01987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33170/0 (socket says 172.21.15.111:33170) 2024-09-16T08:59:44.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80ef7fe640 1 -- 172.21.15.111:0/1590265949 learned_addr learned my addr 172.21.15.111:0/1590265949 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80e7fff640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80f0108790 0x7f80f0198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80effff640 1 -- 172.21.15.111:0/1590265949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80f0108790 msgr2=0x7f80f0198d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80effff640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80f0108790 0x7f80f0198d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80effff640 1 -- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f0102d80 msgr2=0x7f80f01987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80effff640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f0102d80 0x7f80f01987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80effff640 1 -- 172.21.15.111:0/1590265949 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f019e8a0 con 0x7f80f0109110 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.684+0000 7f80ef7fe640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f0102d80 0x7f80f01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.685+0000 7f80effff640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80f0109110 0x7f80f019e0d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f80e802aeb0 tx=0x7f80e8003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.685+0000 7f80ed7fa640 1 -- 172.21.15.111:0/1590265949 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8003e10 con 0x7f80f0109110 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.685+0000 7f80ed7fa640 1 -- 172.21.15.111:0/1590265949 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f80e8004490 con 0x7f80f0109110 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.685+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f019eb30 con 0x7f80f0109110 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.685+0000 7f80ed7fa640 1 -- 172.21.15.111:0/1590265949 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8030ad0 con 0x7f80f0109110 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.685+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f01a5250 con 0x7f80f0109110 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.686+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f0103160 con 0x7f80f0109110 2024-09-16T08:59:44.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.686+0000 7f80ed7fa640 1 -- 172.21.15.111:0/1590265949 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f80e8004640 con 0x7f80f0109110 2024-09-16T08:59:44.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.687+0000 7f80ed7fa640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f80dc07ae10 0x7f80dc07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:44.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.687+0000 7f80ef7fe640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f80dc07ae10 0x7f80dc07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:44.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.687+0000 7f80ed7fa640 1 -- 172.21.15.111:0/1590265949 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7f80e80bf4d0 con 0x7f80f0109110 2024-09-16T08:59:44.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.688+0000 7f80ef7fe640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f80dc07ae10 0x7f80dc07d2d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f80e00104e0 tx=0x7f80e000b410 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:44.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.692+0000 7f80ed7fa640 1 -- 172.21.15.111:0/1590265949 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f80e808a910 con 0x7f80f0109110 2024-09-16T08:59:44.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.866+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 --> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f80f0063590 con 0x7f80dc07ae10 2024-09-16T08:59:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.872+0000 7f80ed7fa640 1 -- 172.21.15.111:0/1590265949 <== mgr.14607 v2:172.21.15.111:6800/1888355313 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346436 (secure 0 0 0) 0x7f80f0063590 con 0x7f80dc07ae10 2024-09-16T08:59:44.876 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:59:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-09-16T08:59:44.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f80dc07ae10 msgr2=0x7f80dc07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f80dc07ae10 0x7f80dc07d2d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f80e00104e0 tx=0x7f80e000b410 comp rx=0 tx=0).stop 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80f0109110 msgr2=0x7f80f019e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80f0109110 0x7f80f019e0d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f80e802aeb0 tx=0x7f80e8003c20 comp rx=0 tx=0).stop 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 shutdown_connections 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f80dc07ae10 0x7f80dc07d2d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80f0109110 0x7f80f019e0d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80f0108790 0x7f80f0198d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 --2- 172.21.15.111:0/1590265949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f0102d80 0x7f80f01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.875+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 >> 172.21.15.111:0/1590265949 conn(0x7f80f00fe7d0 msgr2=0x7f80f0100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.876+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 shutdown_connections 2024-09-16T08:59:44.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:44.876+0000 7f80f616f640 1 -- 172.21.15.111:0/1590265949 wait complete. 2024-09-16T08:59:46.245 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-09-16T08:59:44.454285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":885,"num_read_kb":742,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":235956,"kb_used_data":7260,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725260,"statfs":{"total":767960285184,"available":767718666240,"internally_reserved":0,"allocated":7434240,"data_stored":4068817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":186,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002892"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204641+0000","last_change":"2024-09-16T08:58:59.257450+0000","last_active":"2024-09-16T08:59:31.204641+0000","last_peered":"2024-09-16T08:59:31.204641+0000","last_clean":"2024-09-16T08:59:31.204641+0000","last_became_active":"2024-09-16T08:58:59.257177+0000","last_became_peered":"2024-09-16T08:58:59.257177+0000","last_unstale":"2024-09-16T08:59:31.204641+0000","last_undegraded":"2024-09-16T08:59:31.204641+0000","last_fullsized":"2024-09-16T08:59:31.204641+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:08:16.833508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203163+0000","last_change":"2024-09-16T08:59:01.266150+0000","last_active":"2024-09-16T08:59:31.203163+0000","last_peered":"2024-09-16T08:59:31.203163+0000","last_clean":"2024-09-16T08:59:31.203163+0000","last_became_active":"2024-09-16T08:59:01.265521+0000","last_became_peered":"2024-09-16T08:59:01.265521+0000","last_unstale":"2024-09-16T08:59:31.203163+0000","last_undegraded":"2024-09-16T08:59:31.203163+0000","last_fullsized":"2024-09-16T08:59:31.203163+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:28.698635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439081+0000","last_change":"2024-09-16T08:59:03.284173+0000","last_active":"2024-09-16T08:59:30.439081+0000","last_peered":"2024-09-16T08:59:30.439081+0000","last_clean":"2024-09-16T08:59:30.439081+0000","last_became_active":"2024-09-16T08:59:03.283877+0000","last_became_peered":"2024-09-16T08:59:03.283877+0000","last_unstale":"2024-09-16T08:59:30.439081+0000","last_undegraded":"2024-09-16T08:59:30.439081+0000","last_fullsized":"2024-09-16T08:59:30.439081+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:53.510657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203491+0000","last_change":"2024-09-16T08:59:05.289579+0000","last_active":"2024-09-16T08:59:31.203491+0000","last_peered":"2024-09-16T08:59:31.203491+0000","last_clean":"2024-09-16T08:59:31.203491+0000","last_became_active":"2024-09-16T08:59:05.289266+0000","last_became_peered":"2024-09-16T08:59:05.289266+0000","last_unstale":"2024-09-16T08:59:31.203491+0000","last_undegraded":"2024-09-16T08:59:31.203491+0000","last_fullsized":"2024-09-16T08:59:31.203491+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:01:50.876283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440642+0000","last_change":"2024-09-16T08:59:05.291614+0000","last_active":"2024-09-16T08:59:30.440642+0000","last_peered":"2024-09-16T08:59:30.440642+0000","last_clean":"2024-09-16T08:59:30.440642+0000","last_became_active":"2024-09-16T08:59:05.291295+0000","last_became_peered":"2024-09-16T08:59:05.291295+0000","last_unstale":"2024-09-16T08:59:30.440642+0000","last_undegraded":"2024-09-16T08:59:30.440642+0000","last_fullsized":"2024-09-16T08:59:30.440642+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:44:58.709337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441188+0000","last_change":"2024-09-16T08:58:59.262723+0000","last_active":"2024-09-16T08:59:30.441188+0000","last_peered":"2024-09-16T08:59:30.441188+0000","last_clean":"2024-09-16T08:59:30.441188+0000","last_became_active":"2024-09-16T08:58:59.262427+0000","last_became_peered":"2024-09-16T08:58:59.262427+0000","last_unstale":"2024-09-16T08:59:30.441188+0000","last_undegraded":"2024-09-16T08:59:30.441188+0000","last_fullsized":"2024-09-16T08:59:30.441188+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:33:20.565876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"61'15","reported_seq":51,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440436+0000","last_change":"2024-09-16T08:59:01.272595+0000","last_active":"2024-09-16T08:59:30.440436+0000","last_peered":"2024-09-16T08:59:30.440436+0000","last_clean":"2024-09-16T08:59:30.440436+0000","last_became_active":"2024-09-16T08:59:01.272304+0000","last_became_peered":"2024-09-16T08:59:01.272304+0000","last_unstale":"2024-09-16T08:59:30.440436+0000","last_undegraded":"2024-09-16T08:59:30.440436+0000","last_fullsized":"2024-09-16T08:59:30.440436+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:55:10.089748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203058+0000","last_change":"2024-09-16T08:59:03.285713+0000","last_active":"2024-09-16T08:59:31.203058+0000","last_peered":"2024-09-16T08:59:31.203058+0000","last_clean":"2024-09-16T08:59:31.203058+0000","last_became_active":"2024-09-16T08:59:03.285390+0000","last_became_peered":"2024-09-16T08:59:03.285390+0000","last_unstale":"2024-09-16T08:59:31.203058+0000","last_undegraded":"2024-09-16T08:59:31.203058+0000","last_fullsized":"2024-09-16T08:59:31.203058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:51.755516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775907+0000","last_change":"2024-09-16T08:58:59.263378+0000","last_active":"2024-09-16T08:59:30.775907+0000","last_peered":"2024-09-16T08:59:30.775907+0000","last_clean":"2024-09-16T08:59:30.775907+0000","last_became_active":"2024-09-16T08:58:59.263094+0000","last_became_peered":"2024-09-16T08:58:59.263094+0000","last_unstale":"2024-09-16T08:59:30.775907+0000","last_undegraded":"2024-09-16T08:59:30.775907+0000","last_fullsized":"2024-09-16T08:59:30.775907+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:15.702764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203704+0000","last_change":"2024-09-16T08:59:01.266657+0000","last_active":"2024-09-16T08:59:31.203704+0000","last_peered":"2024-09-16T08:59:31.203704+0000","last_clean":"2024-09-16T08:59:31.203704+0000","last_became_active":"2024-09-16T08:59:01.266349+0000","last_became_peered":"2024-09-16T08:59:01.266349+0000","last_unstale":"2024-09-16T08:59:31.203704+0000","last_undegraded":"2024-09-16T08:59:31.203704+0000","last_fullsized":"2024-09-16T08:59:31.203704+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:11.945878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775547+0000","last_change":"2024-09-16T08:59:03.281873+0000","last_active":"2024-09-16T08:59:30.775547+0000","last_peered":"2024-09-16T08:59:30.775547+0000","last_clean":"2024-09-16T08:59:30.775547+0000","last_became_active":"2024-09-16T08:59:03.281541+0000","last_became_peered":"2024-09-16T08:59:03.281541+0000","last_unstale":"2024-09-16T08:59:30.775547+0000","last_undegraded":"2024-09-16T08:59:30.775547+0000","last_fullsized":"2024-09-16T08:59:30.775547+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:04:02.903335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203263+0000","last_change":"2024-09-16T08:59:05.293990+0000","last_active":"2024-09-16T08:59:31.203263+0000","last_peered":"2024-09-16T08:59:31.203263+0000","last_clean":"2024-09-16T08:59:31.203263+0000","last_became_active":"2024-09-16T08:59:05.293648+0000","last_became_peered":"2024-09-16T08:59:05.293648+0000","last_unstale":"2024-09-16T08:59:31.203263+0000","last_undegraded":"2024-09-16T08:59:31.203263+0000","last_fullsized":"2024-09-16T08:59:31.203263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:10:42.342067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775773+0000","last_change":"2024-09-16T08:58:59.260319+0000","last_active":"2024-09-16T08:59:30.775773+0000","last_peered":"2024-09-16T08:59:30.775773+0000","last_clean":"2024-09-16T08:59:30.775773+0000","last_became_active":"2024-09-16T08:58:59.260043+0000","last_became_peered":"2024-09-16T08:58:59.260043+0000","last_unstale":"2024-09-16T08:59:30.775773+0000","last_undegraded":"2024-09-16T08:59:30.775773+0000","last_fullsized":"2024-09-16T08:59:30.775773+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:11.800206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"61'5","reported_seq":36,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203050+0000","last_change":"2024-09-16T08:59:01.266539+0000","last_active":"2024-09-16T08:59:31.203050+0000","last_peered":"2024-09-16T08:59:31.203050+0000","last_clean":"2024-09-16T08:59:31.203050+0000","last_became_active":"2024-09-16T08:59:01.265489+0000","last_became_peered":"2024-09-16T08:59:01.265489+0000","last_unstale":"2024-09-16T08:59:31.203050+0000","last_undegraded":"2024-09-16T08:59:31.203050+0000","last_fullsized":"2024-09-16T08:59:31.203050+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:07.428651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440512+0000","last_change":"2024-09-16T08:59:03.286289+0000","last_active":"2024-09-16T08:59:30.440512+0000","last_peered":"2024-09-16T08:59:30.440512+0000","last_clean":"2024-09-16T08:59:30.440512+0000","last_became_active":"2024-09-16T08:59:03.285037+0000","last_became_peered":"2024-09-16T08:59:03.285037+0000","last_unstale":"2024-09-16T08:59:30.440512+0000","last_undegraded":"2024-09-16T08:59:30.440512+0000","last_fullsized":"2024-09-16T08:59:30.440512+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:35.283984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775404+0000","last_change":"2024-09-16T08:59:05.292585+0000","last_active":"2024-09-16T08:59:30.775404+0000","last_peered":"2024-09-16T08:59:30.775404+0000","last_clean":"2024-09-16T08:59:30.775404+0000","last_became_active":"2024-09-16T08:59:05.291469+0000","last_became_peered":"2024-09-16T08:59:05.291469+0000","last_unstale":"2024-09-16T08:59:30.775404+0000","last_undegraded":"2024-09-16T08:59:30.775404+0000","last_fullsized":"2024-09-16T08:59:30.775404+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:43:12.870803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204212+0000","last_change":"2024-09-16T08:59:05.287502+0000","last_active":"2024-09-16T08:59:31.204212+0000","last_peered":"2024-09-16T08:59:31.204212+0000","last_clean":"2024-09-16T08:59:31.204212+0000","last_became_active":"2024-09-16T08:59:05.287231+0000","last_became_peered":"2024-09-16T08:59:05.287231+0000","last_unstale":"2024-09-16T08:59:31.204212+0000","last_undegraded":"2024-09-16T08:59:31.204212+0000","last_fullsized":"2024-09-16T08:59:31.204212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:15.074348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204895+0000","last_change":"2024-09-16T08:58:59.260923+0000","last_active":"2024-09-16T08:59:31.204895+0000","last_peered":"2024-09-16T08:59:31.204895+0000","last_clean":"2024-09-16T08:59:31.204895+0000","last_became_active":"2024-09-16T08:58:59.260638+0000","last_became_peered":"2024-09-16T08:58:59.260638+0000","last_unstale":"2024-09-16T08:59:31.204895+0000","last_undegraded":"2024-09-16T08:59:31.204895+0000","last_fullsized":"2024-09-16T08:59:31.204895+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:49.994022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"61'15","reported_seq":51,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781962+0000","last_change":"2024-09-16T08:59:01.271728+0000","last_active":"2024-09-16T08:59:30.781962+0000","last_peered":"2024-09-16T08:59:30.781962+0000","last_clean":"2024-09-16T08:59:30.781962+0000","last_became_active":"2024-09-16T08:59:01.271428+0000","last_became_peered":"2024-09-16T08:59:01.271428+0000","last_unstale":"2024-09-16T08:59:30.781962+0000","last_undegraded":"2024-09-16T08:59:30.781962+0000","last_fullsized":"2024-09-16T08:59:30.781962+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:57.801373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440183+0000","last_change":"2024-09-16T08:59:03.284033+0000","last_active":"2024-09-16T08:59:30.440183+0000","last_peered":"2024-09-16T08:59:30.440183+0000","last_clean":"2024-09-16T08:59:30.440183+0000","last_became_active":"2024-09-16T08:59:03.283603+0000","last_became_peered":"2024-09-16T08:59:03.283603+0000","last_unstale":"2024-09-16T08:59:30.440183+0000","last_undegraded":"2024-09-16T08:59:30.440183+0000","last_fullsized":"2024-09-16T08:59:30.440183+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:58:49.047897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441203+0000","last_change":"2024-09-16T08:59:05.297491+0000","last_active":"2024-09-16T08:59:30.441203+0000","last_peered":"2024-09-16T08:59:30.441203+0000","last_clean":"2024-09-16T08:59:30.441203+0000","last_became_active":"2024-09-16T08:59:05.297022+0000","last_became_peered":"2024-09-16T08:59:05.297022+0000","last_unstale":"2024-09-16T08:59:30.441203+0000","last_undegraded":"2024-09-16T08:59:30.441203+0000","last_fullsized":"2024-09-16T08:59:30.441203+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:34:04.684922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"54'1","reported_seq":36,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203823+0000","last_change":"2024-09-16T08:58:59.256620+0000","last_active":"2024-09-16T08:59:31.203823+0000","last_peered":"2024-09-16T08:59:31.203823+0000","last_clean":"2024-09-16T08:59:31.203823+0000","last_became_active":"2024-09-16T08:58:59.256268+0000","last_became_peered":"2024-09-16T08:58:59.256268+0000","last_unstale":"2024-09-16T08:59:31.203823+0000","last_undegraded":"2024-09-16T08:59:31.203823+0000","last_fullsized":"2024-09-16T08:59:31.203823+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:10.992783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"61'12","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440866+0000","last_change":"2024-09-16T08:59:01.269703+0000","last_active":"2024-09-16T08:59:30.440866+0000","last_peered":"2024-09-16T08:59:30.440866+0000","last_clean":"2024-09-16T08:59:30.440866+0000","last_became_active":"2024-09-16T08:59:01.269373+0000","last_became_peered":"2024-09-16T08:59:01.269373+0000","last_unstale":"2024-09-16T08:59:30.440866+0000","last_undegraded":"2024-09-16T08:59:30.440866+0000","last_fullsized":"2024-09-16T08:59:30.440866+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:10.213444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203980+0000","last_change":"2024-09-16T08:59:03.273760+0000","last_active":"2024-09-16T08:59:31.203980+0000","last_peered":"2024-09-16T08:59:31.203980+0000","last_clean":"2024-09-16T08:59:31.203980+0000","last_became_active":"2024-09-16T08:59:03.273475+0000","last_became_peered":"2024-09-16T08:59:03.273475+0000","last_unstale":"2024-09-16T08:59:31.203980+0000","last_undegraded":"2024-09-16T08:59:31.203980+0000","last_fullsized":"2024-09-16T08:59:31.203980+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:42:32.472066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"61'1","reported_seq":19,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440465+0000","last_change":"2024-09-16T08:59:05.295383+0000","last_active":"2024-09-16T08:59:30.440465+0000","last_peered":"2024-09-16T08:59:30.440465+0000","last_clean":"2024-09-16T08:59:30.440465+0000","last_became_active":"2024-09-16T08:59:05.295028+0000","last_became_peered":"2024-09-16T08:59:05.295028+0000","last_unstale":"2024-09-16T08:59:30.440465+0000","last_undegraded":"2024-09-16T08:59:30.440465+0000","last_fullsized":"2024-09-16T08:59:30.440465+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:06:24.957000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"54'1","reported_seq":31,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439792+0000","last_change":"2024-09-16T08:58:59.259592+0000","last_active":"2024-09-16T08:59:30.439792+0000","last_peered":"2024-09-16T08:59:30.439792+0000","last_clean":"2024-09-16T08:59:30.439792+0000","last_became_active":"2024-09-16T08:58:59.259298+0000","last_became_peered":"2024-09-16T08:58:59.259298+0000","last_unstale":"2024-09-16T08:59:30.439792+0000","last_undegraded":"2024-09-16T08:59:30.439792+0000","last_fullsized":"2024-09-16T08:59:30.439792+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:09:13.136494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"61'10","reported_seq":41,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204250+0000","last_change":"2024-09-16T08:59:01.435269+0000","last_active":"2024-09-16T08:59:31.204250+0000","last_peered":"2024-09-16T08:59:31.204250+0000","last_clean":"2024-09-16T08:59:31.204250+0000","last_became_active":"2024-09-16T08:59:01.434961+0000","last_became_peered":"2024-09-16T08:59:01.434961+0000","last_unstale":"2024-09-16T08:59:31.204250+0000","last_undegraded":"2024-09-16T08:59:31.204250+0000","last_fullsized":"2024-09-16T08:59:31.204250+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:03.566813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"61'8","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318219+0000","last_change":"2024-09-16T08:59:03.283645+0000","last_active":"2024-09-16T08:59:34.318219+0000","last_peered":"2024-09-16T08:59:34.318219+0000","last_clean":"2024-09-16T08:59:34.318219+0000","last_became_active":"2024-09-16T08:59:03.282895+0000","last_became_peered":"2024-09-16T08:59:03.282895+0000","last_unstale":"2024-09-16T08:59:34.318219+0000","last_undegraded":"2024-09-16T08:59:34.318219+0000","last_fullsized":"2024-09-16T08:59:34.318219+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:31.346405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"61'15","reported_seq":51,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441525+0000","last_change":"2024-09-16T08:59:01.273582+0000","last_active":"2024-09-16T08:59:30.441525+0000","last_peered":"2024-09-16T08:59:30.441525+0000","last_clean":"2024-09-16T08:59:30.441525+0000","last_became_active":"2024-09-16T08:59:01.273304+0000","last_became_peered":"2024-09-16T08:59:01.273304+0000","last_unstale":"2024-09-16T08:59:30.441525+0000","last_undegraded":"2024-09-16T08:59:30.441525+0000","last_fullsized":"2024-09-16T08:59:30.441525+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:07.272672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440545+0000","last_change":"2024-09-16T08:58:59.258366+0000","last_active":"2024-09-16T08:59:30.440545+0000","last_peered":"2024-09-16T08:59:30.440545+0000","last_clean":"2024-09-16T08:59:30.440545+0000","last_became_active":"2024-09-16T08:58:59.258088+0000","last_became_peered":"2024-09-16T08:58:59.258088+0000","last_unstale":"2024-09-16T08:59:30.440545+0000","last_undegraded":"2024-09-16T08:59:30.440545+0000","last_fullsized":"2024-09-16T08:59:30.440545+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:39:28.370346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318204+0000","last_change":"2024-09-16T08:59:03.278461+0000","last_active":"2024-09-16T08:59:34.318204+0000","last_peered":"2024-09-16T08:59:34.318204+0000","last_clean":"2024-09-16T08:59:34.318204+0000","last_became_active":"2024-09-16T08:59:03.278191+0000","last_became_peered":"2024-09-16T08:59:03.278191+0000","last_unstale":"2024-09-16T08:59:34.318204+0000","last_undegraded":"2024-09-16T08:59:34.318204+0000","last_fullsized":"2024-09-16T08:59:34.318204+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:52:27.612755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775636+0000","last_change":"2024-09-16T08:59:05.292173+0000","last_active":"2024-09-16T08:59:30.775636+0000","last_peered":"2024-09-16T08:59:30.775636+0000","last_clean":"2024-09-16T08:59:30.775636+0000","last_became_active":"2024-09-16T08:59:05.291813+0000","last_became_peered":"2024-09-16T08:59:05.291813+0000","last_unstale":"2024-09-16T08:59:30.775636+0000","last_undegraded":"2024-09-16T08:59:30.775636+0000","last_fullsized":"2024-09-16T08:59:30.775636+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:52:06.061769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"61'18","reported_seq":58,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440221+0000","last_change":"2024-09-16T08:59:01.433916+0000","last_active":"2024-09-16T08:59:30.440221+0000","last_peered":"2024-09-16T08:59:30.440221+0000","last_clean":"2024-09-16T08:59:30.440221+0000","last_became_active":"2024-09-16T08:59:01.433358+0000","last_became_peered":"2024-09-16T08:59:01.433358+0000","last_unstale":"2024-09-16T08:59:30.440221+0000","last_undegraded":"2024-09-16T08:59:30.440221+0000","last_fullsized":"2024-09-16T08:59:30.440221+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:13.471300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441895+0000","last_change":"2024-09-16T08:58:59.264510+0000","last_active":"2024-09-16T08:59:30.441895+0000","last_peered":"2024-09-16T08:59:30.441895+0000","last_clean":"2024-09-16T08:59:30.441895+0000","last_became_active":"2024-09-16T08:58:59.264199+0000","last_became_peered":"2024-09-16T08:58:59.264199+0000","last_unstale":"2024-09-16T08:59:30.441895+0000","last_undegraded":"2024-09-16T08:59:30.441895+0000","last_fullsized":"2024-09-16T08:59:30.441895+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:32.005141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203160+0000","last_change":"2024-09-16T08:59:03.280701+0000","last_active":"2024-09-16T08:59:31.203160+0000","last_peered":"2024-09-16T08:59:31.203160+0000","last_clean":"2024-09-16T08:59:31.203160+0000","last_became_active":"2024-09-16T08:59:03.280440+0000","last_became_peered":"2024-09-16T08:59:03.280440+0000","last_unstale":"2024-09-16T08:59:31.203160+0000","last_undegraded":"2024-09-16T08:59:31.203160+0000","last_fullsized":"2024-09-16T08:59:31.203160+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:21:52.705170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203681+0000","last_change":"2024-09-16T08:59:05.288877+0000","last_active":"2024-09-16T08:59:31.203681+0000","last_peered":"2024-09-16T08:59:31.203681+0000","last_clean":"2024-09-16T08:59:31.203681+0000","last_became_active":"2024-09-16T08:59:05.288570+0000","last_became_peered":"2024-09-16T08:59:05.288570+0000","last_unstale":"2024-09-16T08:59:31.203681+0000","last_undegraded":"2024-09-16T08:59:31.203681+0000","last_fullsized":"2024-09-16T08:59:31.203681+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:59.393078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"61'14","reported_seq":47,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203368+0000","last_change":"2024-09-16T08:59:01.264825+0000","last_active":"2024-09-16T08:59:31.203368+0000","last_peered":"2024-09-16T08:59:31.203368+0000","last_clean":"2024-09-16T08:59:31.203368+0000","last_became_active":"2024-09-16T08:59:01.264064+0000","last_became_peered":"2024-09-16T08:59:01.264064+0000","last_unstale":"2024-09-16T08:59:31.203368+0000","last_undegraded":"2024-09-16T08:59:31.203368+0000","last_fullsized":"2024-09-16T08:59:31.203368+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:23.056600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"54'1","reported_seq":31,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204793+0000","last_change":"2024-09-16T08:58:59.259100+0000","last_active":"2024-09-16T08:59:31.204793+0000","last_peered":"2024-09-16T08:59:31.204793+0000","last_clean":"2024-09-16T08:59:31.204793+0000","last_became_active":"2024-09-16T08:58:59.258795+0000","last_became_peered":"2024-09-16T08:58:59.258795+0000","last_unstale":"2024-09-16T08:59:31.204793+0000","last_undegraded":"2024-09-16T08:59:31.204793+0000","last_fullsized":"2024-09-16T08:59:31.204793+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:39.926579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318349+0000","last_change":"2024-09-16T08:59:03.280018+0000","last_active":"2024-09-16T08:59:34.318349+0000","last_peered":"2024-09-16T08:59:34.318349+0000","last_clean":"2024-09-16T08:59:34.318349+0000","last_became_active":"2024-09-16T08:59:03.279718+0000","last_became_peered":"2024-09-16T08:59:03.279718+0000","last_unstale":"2024-09-16T08:59:34.318349+0000","last_undegraded":"2024-09-16T08:59:34.318349+0000","last_fullsized":"2024-09-16T08:59:34.318349+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:37.177323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440543+0000","last_change":"2024-09-16T08:59:05.295105+0000","last_active":"2024-09-16T08:59:30.440543+0000","last_peered":"2024-09-16T08:59:30.440543+0000","last_clean":"2024-09-16T08:59:30.440543+0000","last_became_active":"2024-09-16T08:59:05.294831+0000","last_became_peered":"2024-09-16T08:59:05.294831+0000","last_unstale":"2024-09-16T08:59:30.440543+0000","last_undegraded":"2024-09-16T08:59:30.440543+0000","last_fullsized":"2024-09-16T08:59:30.440543+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:36:03.315109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"61'10","reported_seq":41,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439429+0000","last_change":"2024-09-16T08:59:01.267284+0000","last_active":"2024-09-16T08:59:30.439429+0000","last_peered":"2024-09-16T08:59:30.439429+0000","last_clean":"2024-09-16T08:59:30.439429+0000","last_became_active":"2024-09-16T08:59:01.266969+0000","last_became_peered":"2024-09-16T08:59:01.266969+0000","last_unstale":"2024-09-16T08:59:30.439429+0000","last_undegraded":"2024-09-16T08:59:30.439429+0000","last_fullsized":"2024-09-16T08:59:30.439429+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:55:10.837684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775949+0000","last_change":"2024-09-16T08:58:59.256998+0000","last_active":"2024-09-16T08:59:30.775949+0000","last_peered":"2024-09-16T08:59:30.775949+0000","last_clean":"2024-09-16T08:59:30.775949+0000","last_became_active":"2024-09-16T08:58:59.256705+0000","last_became_peered":"2024-09-16T08:58:59.256705+0000","last_unstale":"2024-09-16T08:59:30.775949+0000","last_undegraded":"2024-09-16T08:59:30.775949+0000","last_fullsized":"2024-09-16T08:59:30.775949+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:14:43.704274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"61'8","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318266+0000","last_change":"2024-09-16T08:59:03.286630+0000","last_active":"2024-09-16T08:59:34.318266+0000","last_peered":"2024-09-16T08:59:34.318266+0000","last_clean":"2024-09-16T08:59:34.318266+0000","last_became_active":"2024-09-16T08:59:03.286357+0000","last_became_peered":"2024-09-16T08:59:03.286357+0000","last_unstale":"2024-09-16T08:59:34.318266+0000","last_undegraded":"2024-09-16T08:59:34.318266+0000","last_fullsized":"2024-09-16T08:59:34.318266+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:07:29.240876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203414+0000","last_change":"2024-09-16T08:59:05.289940+0000","last_active":"2024-09-16T08:59:31.203414+0000","last_peered":"2024-09-16T08:59:31.203414+0000","last_clean":"2024-09-16T08:59:31.203414+0000","last_became_active":"2024-09-16T08:59:05.289652+0000","last_became_peered":"2024-09-16T08:59:05.289652+0000","last_unstale":"2024-09-16T08:59:31.203414+0000","last_undegraded":"2024-09-16T08:59:31.203414+0000","last_fullsized":"2024-09-16T08:59:31.203414+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:59.679662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"61'19","reported_seq":62,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204625+0000","last_change":"2024-09-16T08:59:01.433867+0000","last_active":"2024-09-16T08:59:31.204625+0000","last_peered":"2024-09-16T08:59:31.204625+0000","last_clean":"2024-09-16T08:59:31.204625+0000","last_became_active":"2024-09-16T08:59:01.433528+0000","last_became_peered":"2024-09-16T08:59:01.433528+0000","last_unstale":"2024-09-16T08:59:31.204625+0000","last_undegraded":"2024-09-16T08:59:31.204625+0000","last_fullsized":"2024-09-16T08:59:31.204625+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:25.700392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439917+0000","last_change":"2024-09-16T08:58:59.258343+0000","last_active":"2024-09-16T08:59:30.439917+0000","last_peered":"2024-09-16T08:59:30.439917+0000","last_clean":"2024-09-16T08:59:30.439917+0000","last_became_active":"2024-09-16T08:58:59.258097+0000","last_became_peered":"2024-09-16T08:58:59.258097+0000","last_unstale":"2024-09-16T08:59:30.439917+0000","last_undegraded":"2024-09-16T08:59:30.439917+0000","last_fullsized":"2024-09-16T08:59:30.439917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:41.283169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203069+0000","last_change":"2024-09-16T08:59:03.283011+0000","last_active":"2024-09-16T08:59:31.203069+0000","last_peered":"2024-09-16T08:59:31.203069+0000","last_clean":"2024-09-16T08:59:31.203069+0000","last_became_active":"2024-09-16T08:59:03.282735+0000","last_became_peered":"2024-09-16T08:59:03.282735+0000","last_unstale":"2024-09-16T08:59:31.203069+0000","last_undegraded":"2024-09-16T08:59:31.203069+0000","last_fullsized":"2024-09-16T08:59:31.203069+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:35.440091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439039+0000","last_change":"2024-09-16T08:59:05.285275+0000","last_active":"2024-09-16T08:59:30.439039+0000","last_peered":"2024-09-16T08:59:30.439039+0000","last_clean":"2024-09-16T08:59:30.439039+0000","last_became_active":"2024-09-16T08:59:05.284926+0000","last_became_peered":"2024-09-16T08:59:05.284926+0000","last_unstale":"2024-09-16T08:59:30.439039+0000","last_undegraded":"2024-09-16T08:59:30.439039+0000","last_fullsized":"2024-09-16T08:59:30.439039+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:38.799825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"61'28","reported_seq":83,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318340+0000","last_change":"2024-09-16T08:59:01.273742+0000","last_active":"2024-09-16T08:59:34.318340+0000","last_peered":"2024-09-16T08:59:34.318340+0000","last_clean":"2024-09-16T08:59:34.318340+0000","last_became_active":"2024-09-16T08:59:01.273463+0000","last_became_peered":"2024-09-16T08:59:01.273463+0000","last_unstale":"2024-09-16T08:59:34.318340+0000","last_undegraded":"2024-09-16T08:59:34.318340+0000","last_fullsized":"2024-09-16T08:59:34.318340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:45:46.472915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204050+0000","last_change":"2024-09-16T08:58:59.264820+0000","last_active":"2024-09-16T08:59:31.204050+0000","last_peered":"2024-09-16T08:59:31.204050+0000","last_clean":"2024-09-16T08:59:31.204050+0000","last_became_active":"2024-09-16T08:58:59.264413+0000","last_became_peered":"2024-09-16T08:58:59.264413+0000","last_unstale":"2024-09-16T08:59:31.204050+0000","last_undegraded":"2024-09-16T08:59:31.204050+0000","last_fullsized":"2024-09-16T08:59:31.204050+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:43:27.553191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"56'2","reported_seq":40,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775624+0000","last_change":"2024-09-16T08:59:01.262367+0000","last_active":"2024-09-16T08:59:30.775624+0000","last_peered":"2024-09-16T08:59:30.775624+0000","last_clean":"2024-09-16T08:59:30.775624+0000","last_became_active":"2024-09-16T08:58:58.246232+0000","last_became_peered":"2024-09-16T08:58:58.246232+0000","last_unstale":"2024-09-16T08:59:30.775624+0000","last_undegraded":"2024-09-16T08:59:30.775624+0000","last_fullsized":"2024-09-16T08:59:30.775624+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_clean_scrub_stamp":"2024-09-16T08:58:57.230240+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:29.095207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001635145,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204290+0000","last_change":"2024-09-16T08:59:03.285616+0000","last_active":"2024-09-16T08:59:31.204290+0000","last_peered":"2024-09-16T08:59:31.204290+0000","last_clean":"2024-09-16T08:59:31.204290+0000","last_became_active":"2024-09-16T08:59:03.284205+0000","last_became_peered":"2024-09-16T08:59:03.284205+0000","last_unstale":"2024-09-16T08:59:31.204290+0000","last_undegraded":"2024-09-16T08:59:31.204290+0000","last_fullsized":"2024-09-16T08:59:31.204290+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:18:00.305433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440910+0000","last_change":"2024-09-16T08:59:05.293367+0000","last_active":"2024-09-16T08:59:30.440910+0000","last_peered":"2024-09-16T08:59:30.440910+0000","last_clean":"2024-09-16T08:59:30.440910+0000","last_became_active":"2024-09-16T08:59:05.291800+0000","last_became_peered":"2024-09-16T08:59:05.291800+0000","last_unstale":"2024-09-16T08:59:30.440910+0000","last_undegraded":"2024-09-16T08:59:30.440910+0000","last_fullsized":"2024-09-16T08:59:30.440910+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:49.809698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"61'13","reported_seq":53,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441660+0000","last_change":"2024-09-16T08:59:01.271118+0000","last_active":"2024-09-16T08:59:30.441660+0000","last_peered":"2024-09-16T08:59:30.441660+0000","last_clean":"2024-09-16T08:59:30.441660+0000","last_became_active":"2024-09-16T08:59:01.270835+0000","last_became_peered":"2024-09-16T08:59:01.270835+0000","last_unstale":"2024-09-16T08:59:30.441660+0000","last_undegraded":"2024-09-16T08:59:30.441660+0000","last_fullsized":"2024-09-16T08:59:30.441660+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:30.651659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439285+0000","last_change":"2024-09-16T08:58:59.263658+0000","last_active":"2024-09-16T08:59:30.439285+0000","last_peered":"2024-09-16T08:59:30.439285+0000","last_clean":"2024-09-16T08:59:30.439285+0000","last_became_active":"2024-09-16T08:58:59.263366+0000","last_became_peered":"2024-09-16T08:58:59.263366+0000","last_unstale":"2024-09-16T08:59:30.439285+0000","last_undegraded":"2024-09-16T08:59:30.439285+0000","last_fullsized":"2024-09-16T08:59:30.439285+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:07.710217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"54'1","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781971+0000","last_change":"2024-09-16T08:59:01.263688+0000","last_active":"2024-09-16T08:59:30.781971+0000","last_peered":"2024-09-16T08:59:30.781971+0000","last_clean":"2024-09-16T08:59:30.781971+0000","last_became_active":"2024-09-16T08:58:58.248552+0000","last_became_peered":"2024-09-16T08:58:58.248552+0000","last_unstale":"2024-09-16T08:59:30.781971+0000","last_undegraded":"2024-09-16T08:59:30.781971+0000","last_fullsized":"2024-09-16T08:59:30.781971+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_clean_scrub_stamp":"2024-09-16T08:58:57.230240+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:44.765434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003167675,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781651+0000","last_change":"2024-09-16T08:59:03.280876+0000","last_active":"2024-09-16T08:59:30.781651+0000","last_peered":"2024-09-16T08:59:30.781651+0000","last_clean":"2024-09-16T08:59:30.781651+0000","last_became_active":"2024-09-16T08:59:03.280590+0000","last_became_peered":"2024-09-16T08:59:03.280590+0000","last_unstale":"2024-09-16T08:59:30.781651+0000","last_undegraded":"2024-09-16T08:59:30.781651+0000","last_fullsized":"2024-09-16T08:59:30.781651+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:43.999322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440644+0000","last_change":"2024-09-16T08:59:05.293663+0000","last_active":"2024-09-16T08:59:30.440644+0000","last_peered":"2024-09-16T08:59:30.440644+0000","last_clean":"2024-09-16T08:59:30.440644+0000","last_became_active":"2024-09-16T08:59:05.293366+0000","last_became_peered":"2024-09-16T08:59:05.293366+0000","last_unstale":"2024-09-16T08:59:30.440644+0000","last_undegraded":"2024-09-16T08:59:30.440644+0000","last_fullsized":"2024-09-16T08:59:30.440644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:43.979621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"61'12","reported_seq":44,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204380+0000","last_change":"2024-09-16T08:59:01.272958+0000","last_active":"2024-09-16T08:59:31.204380+0000","last_peered":"2024-09-16T08:59:31.204380+0000","last_clean":"2024-09-16T08:59:31.204380+0000","last_became_active":"2024-09-16T08:59:01.272635+0000","last_became_peered":"2024-09-16T08:59:01.272635+0000","last_unstale":"2024-09-16T08:59:31.204380+0000","last_undegraded":"2024-09-16T08:59:31.204380+0000","last_fullsized":"2024-09-16T08:59:31.204380+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:01:13.616063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203350+0000","last_change":"2024-09-16T08:58:59.259671+0000","last_active":"2024-09-16T08:59:31.203350+0000","last_peered":"2024-09-16T08:59:31.203350+0000","last_clean":"2024-09-16T08:59:31.203350+0000","last_became_active":"2024-09-16T08:58:59.259401+0000","last_became_peered":"2024-09-16T08:58:59.259401+0000","last_unstale":"2024-09-16T08:59:31.203350+0000","last_undegraded":"2024-09-16T08:59:31.203350+0000","last_fullsized":"2024-09-16T08:59:31.203350+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:27:04.456124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:40.088326+0000","last_change":"2024-09-16T08:59:01.433442+0000","last_active":"2024-09-16T08:59:40.088326+0000","last_peered":"2024-09-16T08:59:40.088326+0000","last_clean":"2024-09-16T08:59:40.088326+0000","last_became_active":"2024-09-16T08:58:58.248425+0000","last_became_peered":"2024-09-16T08:58:58.248425+0000","last_unstale":"2024-09-16T08:59:40.088326+0000","last_undegraded":"2024-09-16T08:59:40.088326+0000","last_fullsized":"2024-09-16T08:59:40.088326+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:57.230240+0000","last_clean_scrub_stamp":"2024-09-16T08:58:57.230240+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:13:32.978470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069780000000000005,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":32,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.776001+0000","last_change":"2024-09-16T08:59:03.281201+0000","last_active":"2024-09-16T08:59:30.776001+0000","last_peered":"2024-09-16T08:59:30.776001+0000","last_clean":"2024-09-16T08:59:30.776001+0000","last_became_active":"2024-09-16T08:59:03.280751+0000","last_became_peered":"2024-09-16T08:59:03.280751+0000","last_unstale":"2024-09-16T08:59:30.776001+0000","last_undegraded":"2024-09-16T08:59:30.776001+0000","last_fullsized":"2024-09-16T08:59:30.776001+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:02:08.970051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439478+0000","last_change":"2024-09-16T08:59:05.290351+0000","last_active":"2024-09-16T08:59:30.439478+0000","last_peered":"2024-09-16T08:59:30.439478+0000","last_clean":"2024-09-16T08:59:30.439478+0000","last_became_active":"2024-09-16T08:59:05.289470+0000","last_became_peered":"2024-09-16T08:59:05.289470+0000","last_unstale":"2024-09-16T08:59:30.439478+0000","last_undegraded":"2024-09-16T08:59:30.439478+0000","last_fullsized":"2024-09-16T08:59:30.439478+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:48.796868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"61'16","reported_seq":57,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318308+0000","last_change":"2024-09-16T08:59:01.435018+0000","last_active":"2024-09-16T08:59:34.318308+0000","last_peered":"2024-09-16T08:59:34.318308+0000","last_clean":"2024-09-16T08:59:34.318308+0000","last_became_active":"2024-09-16T08:59:01.434721+0000","last_became_peered":"2024-09-16T08:59:01.434721+0000","last_unstale":"2024-09-16T08:59:34.318308+0000","last_undegraded":"2024-09-16T08:59:34.318308+0000","last_fullsized":"2024-09-16T08:59:34.318308+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:42.846463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440652+0000","last_change":"2024-09-16T08:58:59.263361+0000","last_active":"2024-09-16T08:59:30.440652+0000","last_peered":"2024-09-16T08:59:30.440652+0000","last_clean":"2024-09-16T08:59:30.440652+0000","last_became_active":"2024-09-16T08:58:59.263074+0000","last_became_peered":"2024-09-16T08:58:59.263074+0000","last_unstale":"2024-09-16T08:59:30.440652+0000","last_undegraded":"2024-09-16T08:59:30.440652+0000","last_fullsized":"2024-09-16T08:59:30.440652+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:01.696844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"63'79","reported_seq":67,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:32.763949+0000","last_change":"2024-09-16T08:58:49.325063+0000","last_active":"2024-09-16T08:59:32.763949+0000","last_peered":"2024-09-16T08:59:32.763949+0000","last_clean":"2024-09-16T08:59:32.763949+0000","last_became_active":"2024-09-16T08:58:49.319994+0000","last_became_peered":"2024-09-16T08:58:49.319994+0000","last_unstale":"2024-09-16T08:59:32.763949+0000","last_undegraded":"2024-09-16T08:59:32.763949+0000","last_fullsized":"2024-09-16T08:59:32.763949+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:56:27.881800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:56:27.881800+0000","last_clean_scrub_stamp":"2024-09-16T08:56:27.881800+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:11:17.852949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440659+0000","last_change":"2024-09-16T08:59:03.285822+0000","last_active":"2024-09-16T08:59:30.440659+0000","last_peered":"2024-09-16T08:59:30.440659+0000","last_clean":"2024-09-16T08:59:30.440659+0000","last_became_active":"2024-09-16T08:59:03.284965+0000","last_became_peered":"2024-09-16T08:59:03.284965+0000","last_unstale":"2024-09-16T08:59:30.440659+0000","last_undegraded":"2024-09-16T08:59:30.440659+0000","last_fullsized":"2024-09-16T08:59:30.440659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:49:30.467506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775851+0000","last_change":"2024-09-16T08:59:05.291694+0000","last_active":"2024-09-16T08:59:30.775851+0000","last_peered":"2024-09-16T08:59:30.775851+0000","last_clean":"2024-09-16T08:59:30.775851+0000","last_became_active":"2024-09-16T08:59:05.291287+0000","last_became_peered":"2024-09-16T08:59:05.291287+0000","last_unstale":"2024-09-16T08:59:30.775851+0000","last_undegraded":"2024-09-16T08:59:30.775851+0000","last_fullsized":"2024-09-16T08:59:30.775851+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:50:52.992278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"61'11","reported_seq":45,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204331+0000","last_change":"2024-09-16T08:59:01.266728+0000","last_active":"2024-09-16T08:59:31.204331+0000","last_peered":"2024-09-16T08:59:31.204331+0000","last_clean":"2024-09-16T08:59:31.204331+0000","last_became_active":"2024-09-16T08:59:01.266389+0000","last_became_peered":"2024-09-16T08:59:01.266389+0000","last_unstale":"2024-09-16T08:59:31.204331+0000","last_undegraded":"2024-09-16T08:59:31.204331+0000","last_fullsized":"2024-09-16T08:59:31.204331+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:01:03.005036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203599+0000","last_change":"2024-09-16T08:58:59.260106+0000","last_active":"2024-09-16T08:59:31.203599+0000","last_peered":"2024-09-16T08:59:31.203599+0000","last_clean":"2024-09-16T08:59:31.203599+0000","last_became_active":"2024-09-16T08:58:59.259815+0000","last_became_peered":"2024-09-16T08:58:59.259815+0000","last_unstale":"2024-09-16T08:59:31.203599+0000","last_undegraded":"2024-09-16T08:59:31.203599+0000","last_fullsized":"2024-09-16T08:59:31.203599+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:06.550778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775912+0000","last_change":"2024-09-16T08:59:03.284575+0000","last_active":"2024-09-16T08:59:30.775912+0000","last_peered":"2024-09-16T08:59:30.775912+0000","last_clean":"2024-09-16T08:59:30.775912+0000","last_became_active":"2024-09-16T08:59:03.284214+0000","last_became_peered":"2024-09-16T08:59:03.284214+0000","last_unstale":"2024-09-16T08:59:30.775912+0000","last_undegraded":"2024-09-16T08:59:30.775912+0000","last_fullsized":"2024-09-16T08:59:30.775912+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:37.032833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440134+0000","last_change":"2024-09-16T08:59:05.297367+0000","last_active":"2024-09-16T08:59:30.440134+0000","last_peered":"2024-09-16T08:59:30.440134+0000","last_clean":"2024-09-16T08:59:30.440134+0000","last_became_active":"2024-09-16T08:59:05.297010+0000","last_became_peered":"2024-09-16T08:59:05.297010+0000","last_unstale":"2024-09-16T08:59:30.440134+0000","last_undegraded":"2024-09-16T08:59:30.440134+0000","last_fullsized":"2024-09-16T08:59:30.440134+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:55.243088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"61'17","reported_seq":54,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204265+0000","last_change":"2024-09-16T08:59:01.265603+0000","last_active":"2024-09-16T08:59:31.204265+0000","last_peered":"2024-09-16T08:59:31.204265+0000","last_clean":"2024-09-16T08:59:31.204265+0000","last_became_active":"2024-09-16T08:59:01.265279+0000","last_became_peered":"2024-09-16T08:59:01.265279+0000","last_unstale":"2024-09-16T08:59:31.204265+0000","last_undegraded":"2024-09-16T08:59:31.204265+0000","last_fullsized":"2024-09-16T08:59:31.204265+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:02.792134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203350+0000","last_change":"2024-09-16T08:58:59.259712+0000","last_active":"2024-09-16T08:59:31.203350+0000","last_peered":"2024-09-16T08:59:31.203350+0000","last_clean":"2024-09-16T08:59:31.203350+0000","last_became_active":"2024-09-16T08:58:59.259390+0000","last_became_peered":"2024-09-16T08:58:59.259390+0000","last_unstale":"2024-09-16T08:59:31.203350+0000","last_undegraded":"2024-09-16T08:59:31.203350+0000","last_fullsized":"2024-09-16T08:59:31.203350+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:34.944796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439575+0000","last_change":"2024-09-16T08:59:03.279579+0000","last_active":"2024-09-16T08:59:30.439575+0000","last_peered":"2024-09-16T08:59:30.439575+0000","last_clean":"2024-09-16T08:59:30.439575+0000","last_became_active":"2024-09-16T08:59:03.279283+0000","last_became_peered":"2024-09-16T08:59:03.279283+0000","last_unstale":"2024-09-16T08:59:30.439575+0000","last_undegraded":"2024-09-16T08:59:30.439575+0000","last_fullsized":"2024-09-16T08:59:30.439575+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:56.034243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781818+0000","last_change":"2024-09-16T08:59:05.292792+0000","last_active":"2024-09-16T08:59:30.781818+0000","last_peered":"2024-09-16T08:59:30.781818+0000","last_clean":"2024-09-16T08:59:30.781818+0000","last_became_active":"2024-09-16T08:59:05.292378+0000","last_became_peered":"2024-09-16T08:59:05.292378+0000","last_unstale":"2024-09-16T08:59:30.781818+0000","last_undegraded":"2024-09-16T08:59:30.781818+0000","last_fullsized":"2024-09-16T08:59:30.781818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:09:38.969129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"61'10","reported_seq":41,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204026+0000","last_change":"2024-09-16T08:59:01.264761+0000","last_active":"2024-09-16T08:59:31.204026+0000","last_peered":"2024-09-16T08:59:31.204026+0000","last_clean":"2024-09-16T08:59:31.204026+0000","last_became_active":"2024-09-16T08:59:01.264348+0000","last_became_peered":"2024-09-16T08:59:01.264348+0000","last_unstale":"2024-09-16T08:59:31.204026+0000","last_undegraded":"2024-09-16T08:59:31.204026+0000","last_fullsized":"2024-09-16T08:59:31.204026+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:33.406515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441000+0000","last_change":"2024-09-16T08:58:59.256842+0000","last_active":"2024-09-16T08:59:30.441000+0000","last_peered":"2024-09-16T08:59:30.441000+0000","last_clean":"2024-09-16T08:59:30.441000+0000","last_became_active":"2024-09-16T08:58:59.256544+0000","last_became_peered":"2024-09-16T08:58:59.256544+0000","last_unstale":"2024-09-16T08:59:30.441000+0000","last_undegraded":"2024-09-16T08:59:30.441000+0000","last_fullsized":"2024-09-16T08:59:30.441000+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:42.174926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318247+0000","last_change":"2024-09-16T08:59:03.280982+0000","last_active":"2024-09-16T08:59:34.318247+0000","last_peered":"2024-09-16T08:59:34.318247+0000","last_clean":"2024-09-16T08:59:34.318247+0000","last_became_active":"2024-09-16T08:59:03.280167+0000","last_became_peered":"2024-09-16T08:59:03.280167+0000","last_unstale":"2024-09-16T08:59:34.318247+0000","last_undegraded":"2024-09-16T08:59:34.318247+0000","last_fullsized":"2024-09-16T08:59:34.318247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:05.152692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204197+0000","last_change":"2024-09-16T08:59:05.289377+0000","last_active":"2024-09-16T08:59:31.204197+0000","last_peered":"2024-09-16T08:59:31.204197+0000","last_clean":"2024-09-16T08:59:31.204197+0000","last_became_active":"2024-09-16T08:59:05.289089+0000","last_became_peered":"2024-09-16T08:59:05.289089+0000","last_unstale":"2024-09-16T08:59:31.204197+0000","last_undegraded":"2024-09-16T08:59:31.204197+0000","last_fullsized":"2024-09-16T08:59:31.204197+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:28:54.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204510+0000","last_change":"2024-09-16T08:59:01.274085+0000","last_active":"2024-09-16T08:59:31.204510+0000","last_peered":"2024-09-16T08:59:31.204510+0000","last_clean":"2024-09-16T08:59:31.204510+0000","last_became_active":"2024-09-16T08:59:01.273619+0000","last_became_peered":"2024-09-16T08:59:01.273619+0000","last_unstale":"2024-09-16T08:59:31.204510+0000","last_undegraded":"2024-09-16T08:59:31.204510+0000","last_fullsized":"2024-09-16T08:59:31.204510+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:20:40.339908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775910+0000","last_change":"2024-09-16T08:58:59.263139+0000","last_active":"2024-09-16T08:59:30.775910+0000","last_peered":"2024-09-16T08:59:30.775910+0000","last_clean":"2024-09-16T08:59:30.775910+0000","last_became_active":"2024-09-16T08:58:59.262812+0000","last_became_peered":"2024-09-16T08:58:59.262812+0000","last_unstale":"2024-09-16T08:59:30.775910+0000","last_undegraded":"2024-09-16T08:59:30.775910+0000","last_fullsized":"2024-09-16T08:59:30.775910+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:03.695315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781318+0000","last_change":"2024-09-16T08:59:03.275837+0000","last_active":"2024-09-16T08:59:30.781318+0000","last_peered":"2024-09-16T08:59:30.781318+0000","last_clean":"2024-09-16T08:59:30.781318+0000","last_became_active":"2024-09-16T08:59:03.275550+0000","last_became_peered":"2024-09-16T08:59:03.275550+0000","last_unstale":"2024-09-16T08:59:30.781318+0000","last_undegraded":"2024-09-16T08:59:30.781318+0000","last_fullsized":"2024-09-16T08:59:30.781318+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:43.616278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203251+0000","last_change":"2024-09-16T08:59:05.290834+0000","last_active":"2024-09-16T08:59:31.203251+0000","last_peered":"2024-09-16T08:59:31.203251+0000","last_clean":"2024-09-16T08:59:31.203251+0000","last_became_active":"2024-09-16T08:59:05.290449+0000","last_became_peered":"2024-09-16T08:59:05.290449+0000","last_unstale":"2024-09-16T08:59:31.203251+0000","last_undegraded":"2024-09-16T08:59:31.203251+0000","last_fullsized":"2024-09-16T08:59:31.203251+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:48:51.370372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"61'19","reported_seq":57,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203152+0000","last_change":"2024-09-16T08:59:01.434403+0000","last_active":"2024-09-16T08:59:31.203152+0000","last_peered":"2024-09-16T08:59:31.203152+0000","last_clean":"2024-09-16T08:59:31.203152+0000","last_became_active":"2024-09-16T08:59:01.433965+0000","last_became_peered":"2024-09-16T08:59:01.433965+0000","last_unstale":"2024-09-16T08:59:31.203152+0000","last_undegraded":"2024-09-16T08:59:31.203152+0000","last_fullsized":"2024-09-16T08:59:31.203152+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:30.542133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440989+0000","last_change":"2024-09-16T08:58:59.266874+0000","last_active":"2024-09-16T08:59:30.440989+0000","last_peered":"2024-09-16T08:59:30.440989+0000","last_clean":"2024-09-16T08:59:30.440989+0000","last_became_active":"2024-09-16T08:58:59.266555+0000","last_became_peered":"2024-09-16T08:58:59.266555+0000","last_unstale":"2024-09-16T08:59:30.440989+0000","last_undegraded":"2024-09-16T08:59:30.440989+0000","last_fullsized":"2024-09-16T08:59:30.440989+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:15.315810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781394+0000","last_change":"2024-09-16T08:59:03.281576+0000","last_active":"2024-09-16T08:59:30.781394+0000","last_peered":"2024-09-16T08:59:30.781394+0000","last_clean":"2024-09-16T08:59:30.781394+0000","last_became_active":"2024-09-16T08:59:03.280148+0000","last_became_peered":"2024-09-16T08:59:03.280148+0000","last_unstale":"2024-09-16T08:59:30.781394+0000","last_undegraded":"2024-09-16T08:59:30.781394+0000","last_fullsized":"2024-09-16T08:59:30.781394+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:09:35.787588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440034+0000","last_change":"2024-09-16T08:59:05.289415+0000","last_active":"2024-09-16T08:59:30.440034+0000","last_peered":"2024-09-16T08:59:30.440034+0000","last_clean":"2024-09-16T08:59:30.440034+0000","last_became_active":"2024-09-16T08:59:05.288932+0000","last_became_peered":"2024-09-16T08:59:05.288932+0000","last_unstale":"2024-09-16T08:59:30.440034+0000","last_undegraded":"2024-09-16T08:59:30.440034+0000","last_fullsized":"2024-09-16T08:59:30.440034+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:17:27.315454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"61'12","reported_seq":49,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203246+0000","last_change":"2024-09-16T08:59:01.266118+0000","last_active":"2024-09-16T08:59:31.203246+0000","last_peered":"2024-09-16T08:59:31.203246+0000","last_clean":"2024-09-16T08:59:31.203246+0000","last_became_active":"2024-09-16T08:59:01.264492+0000","last_became_peered":"2024-09-16T08:59:01.264492+0000","last_unstale":"2024-09-16T08:59:31.203246+0000","last_undegraded":"2024-09-16T08:59:31.203246+0000","last_fullsized":"2024-09-16T08:59:31.203246+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:46:25.022599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441869+0000","last_change":"2024-09-16T08:58:59.266794+0000","last_active":"2024-09-16T08:59:30.441869+0000","last_peered":"2024-09-16T08:59:30.441869+0000","last_clean":"2024-09-16T08:59:30.441869+0000","last_became_active":"2024-09-16T08:58:59.266467+0000","last_became_peered":"2024-09-16T08:58:59.266467+0000","last_unstale":"2024-09-16T08:59:30.441869+0000","last_undegraded":"2024-09-16T08:59:30.441869+0000","last_fullsized":"2024-09-16T08:59:30.441869+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:13.704059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781285+0000","last_change":"2024-09-16T08:59:03.281208+0000","last_active":"2024-09-16T08:59:30.781285+0000","last_peered":"2024-09-16T08:59:30.781285+0000","last_clean":"2024-09-16T08:59:30.781285+0000","last_became_active":"2024-09-16T08:59:03.280931+0000","last_became_peered":"2024-09-16T08:59:03.280931+0000","last_unstale":"2024-09-16T08:59:30.781285+0000","last_undegraded":"2024-09-16T08:59:30.781285+0000","last_fullsized":"2024-09-16T08:59:30.781285+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:08:45.251981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440692+0000","last_change":"2024-09-16T08:59:05.290572+0000","last_active":"2024-09-16T08:59:30.440692+0000","last_peered":"2024-09-16T08:59:30.440692+0000","last_clean":"2024-09-16T08:59:30.440692+0000","last_became_active":"2024-09-16T08:59:05.290273+0000","last_became_peered":"2024-09-16T08:59:05.290273+0000","last_unstale":"2024-09-16T08:59:30.440692+0000","last_undegraded":"2024-09-16T08:59:30.440692+0000","last_fullsized":"2024-09-16T08:59:30.440692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:02:52.134683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"61'15","reported_seq":51,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775300+0000","last_change":"2024-09-16T08:59:01.434570+0000","last_active":"2024-09-16T08:59:30.775300+0000","last_peered":"2024-09-16T08:59:30.775300+0000","last_clean":"2024-09-16T08:59:30.775300+0000","last_became_active":"2024-09-16T08:59:01.434251+0000","last_became_peered":"2024-09-16T08:59:01.434251+0000","last_unstale":"2024-09-16T08:59:30.775300+0000","last_undegraded":"2024-09-16T08:59:30.775300+0000","last_fullsized":"2024-09-16T08:59:30.775300+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:17:30.165936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"54'2","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441572+0000","last_change":"2024-09-16T08:58:59.266786+0000","last_active":"2024-09-16T08:59:30.441572+0000","last_peered":"2024-09-16T08:59:30.441572+0000","last_clean":"2024-09-16T08:59:30.441572+0000","last_became_active":"2024-09-16T08:58:59.266445+0000","last_became_peered":"2024-09-16T08:58:59.266445+0000","last_unstale":"2024-09-16T08:59:30.441572+0000","last_undegraded":"2024-09-16T08:59:30.441572+0000","last_fullsized":"2024-09-16T08:59:30.441572+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:53:46.828836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318212+0000","last_change":"2024-09-16T08:59:03.283746+0000","last_active":"2024-09-16T08:59:34.318212+0000","last_peered":"2024-09-16T08:59:34.318212+0000","last_clean":"2024-09-16T08:59:34.318212+0000","last_became_active":"2024-09-16T08:59:03.283346+0000","last_became_peered":"2024-09-16T08:59:03.283346+0000","last_unstale":"2024-09-16T08:59:34.318212+0000","last_undegraded":"2024-09-16T08:59:34.318212+0000","last_fullsized":"2024-09-16T08:59:34.318212+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:39.352949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775405+0000","last_change":"2024-09-16T08:59:05.292279+0000","last_active":"2024-09-16T08:59:30.775405+0000","last_peered":"2024-09-16T08:59:30.775405+0000","last_clean":"2024-09-16T08:59:30.775405+0000","last_became_active":"2024-09-16T08:59:05.291768+0000","last_became_peered":"2024-09-16T08:59:05.291768+0000","last_unstale":"2024-09-16T08:59:30.775405+0000","last_undegraded":"2024-09-16T08:59:30.775405+0000","last_fullsized":"2024-09-16T08:59:30.775405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:26.003718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.202800+0000","last_change":"2024-09-16T08:58:59.259063+0000","last_active":"2024-09-16T08:59:31.202800+0000","last_peered":"2024-09-16T08:59:31.202800+0000","last_clean":"2024-09-16T08:59:31.202800+0000","last_became_active":"2024-09-16T08:58:59.258752+0000","last_became_peered":"2024-09-16T08:58:59.258752+0000","last_unstale":"2024-09-16T08:59:31.202800+0000","last_undegraded":"2024-09-16T08:59:31.202800+0000","last_fullsized":"2024-09-16T08:59:31.202800+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:33:46.685367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"61'6","reported_seq":35,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441432+0000","last_change":"2024-09-16T08:59:01.434607+0000","last_active":"2024-09-16T08:59:30.441432+0000","last_peered":"2024-09-16T08:59:30.441432+0000","last_clean":"2024-09-16T08:59:30.441432+0000","last_became_active":"2024-09-16T08:59:01.434292+0000","last_became_peered":"2024-09-16T08:59:01.434292+0000","last_unstale":"2024-09-16T08:59:30.441432+0000","last_undegraded":"2024-09-16T08:59:30.441432+0000","last_fullsized":"2024-09-16T08:59:30.441432+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:09.549373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775780+0000","last_change":"2024-09-16T08:59:03.281523+0000","last_active":"2024-09-16T08:59:30.775780+0000","last_peered":"2024-09-16T08:59:30.775780+0000","last_clean":"2024-09-16T08:59:30.775780+0000","last_became_active":"2024-09-16T08:59:03.280220+0000","last_became_peered":"2024-09-16T08:59:03.280220+0000","last_unstale":"2024-09-16T08:59:30.775780+0000","last_undegraded":"2024-09-16T08:59:30.775780+0000","last_fullsized":"2024-09-16T08:59:30.775780+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:20:36.292718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440663+0000","last_change":"2024-09-16T08:59:05.292037+0000","last_active":"2024-09-16T08:59:30.440663+0000","last_peered":"2024-09-16T08:59:30.440663+0000","last_clean":"2024-09-16T08:59:30.440663+0000","last_became_active":"2024-09-16T08:59:05.291641+0000","last_became_peered":"2024-09-16T08:59:05.291641+0000","last_unstale":"2024-09-16T08:59:30.440663+0000","last_undegraded":"2024-09-16T08:59:30.440663+0000","last_fullsized":"2024-09-16T08:59:30.440663+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:00.620413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204540+0000","last_change":"2024-09-16T08:59:01.433875+0000","last_active":"2024-09-16T08:59:31.204540+0000","last_peered":"2024-09-16T08:59:31.204540+0000","last_clean":"2024-09-16T08:59:31.204540+0000","last_became_active":"2024-09-16T08:59:01.433570+0000","last_became_peered":"2024-09-16T08:59:01.433570+0000","last_unstale":"2024-09-16T08:59:31.204540+0000","last_undegraded":"2024-09-16T08:59:31.204540+0000","last_fullsized":"2024-09-16T08:59:31.204540+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:01:07.210788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440449+0000","last_change":"2024-09-16T08:58:59.265895+0000","last_active":"2024-09-16T08:59:30.440449+0000","last_peered":"2024-09-16T08:59:30.440449+0000","last_clean":"2024-09-16T08:59:30.440449+0000","last_became_active":"2024-09-16T08:58:59.265588+0000","last_became_peered":"2024-09-16T08:58:59.265588+0000","last_unstale":"2024-09-16T08:59:30.440449+0000","last_undegraded":"2024-09-16T08:59:30.440449+0000","last_fullsized":"2024-09-16T08:59:30.440449+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:57:40.324659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441286+0000","last_change":"2024-09-16T08:59:03.282785+0000","last_active":"2024-09-16T08:59:30.441286+0000","last_peered":"2024-09-16T08:59:30.441286+0000","last_clean":"2024-09-16T08:59:30.441286+0000","last_became_active":"2024-09-16T08:59:03.282524+0000","last_became_peered":"2024-09-16T08:59:03.282524+0000","last_unstale":"2024-09-16T08:59:30.441286+0000","last_undegraded":"2024-09-16T08:59:30.441286+0000","last_fullsized":"2024-09-16T08:59:30.441286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:31.268238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.781599+0000","last_change":"2024-09-16T08:59:05.291726+0000","last_active":"2024-09-16T08:59:30.781599+0000","last_peered":"2024-09-16T08:59:30.781599+0000","last_clean":"2024-09-16T08:59:30.781599+0000","last_became_active":"2024-09-16T08:59:05.291390+0000","last_became_peered":"2024-09-16T08:59:05.291390+0000","last_unstale":"2024-09-16T08:59:30.781599+0000","last_undegraded":"2024-09-16T08:59:30.781599+0000","last_fullsized":"2024-09-16T08:59:30.781599+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:49:50.012693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775829+0000","last_change":"2024-09-16T08:59:01.433869+0000","last_active":"2024-09-16T08:59:30.775829+0000","last_peered":"2024-09-16T08:59:30.775829+0000","last_clean":"2024-09-16T08:59:30.775829+0000","last_became_active":"2024-09-16T08:59:01.433531+0000","last_became_peered":"2024-09-16T08:59:01.433531+0000","last_unstale":"2024-09-16T08:59:30.775829+0000","last_undegraded":"2024-09-16T08:59:30.775829+0000","last_fullsized":"2024-09-16T08:59:30.775829+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:32:51.504027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203901+0000","last_change":"2024-09-16T08:58:59.260819+0000","last_active":"2024-09-16T08:59:31.203901+0000","last_peered":"2024-09-16T08:59:31.203901+0000","last_clean":"2024-09-16T08:59:31.203901+0000","last_became_active":"2024-09-16T08:58:59.260293+0000","last_became_peered":"2024-09-16T08:58:59.260293+0000","last_unstale":"2024-09-16T08:59:31.203901+0000","last_undegraded":"2024-09-16T08:59:31.203901+0000","last_fullsized":"2024-09-16T08:59:31.203901+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:53:11.123625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318372+0000","last_change":"2024-09-16T08:59:03.281887+0000","last_active":"2024-09-16T08:59:34.318372+0000","last_peered":"2024-09-16T08:59:34.318372+0000","last_clean":"2024-09-16T08:59:34.318372+0000","last_became_active":"2024-09-16T08:59:03.281598+0000","last_became_peered":"2024-09-16T08:59:03.281598+0000","last_unstale":"2024-09-16T08:59:34.318372+0000","last_undegraded":"2024-09-16T08:59:34.318372+0000","last_fullsized":"2024-09-16T08:59:34.318372+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:10:52.288798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204400+0000","last_change":"2024-09-16T08:59:05.292678+0000","last_active":"2024-09-16T08:59:31.204400+0000","last_peered":"2024-09-16T08:59:31.204400+0000","last_clean":"2024-09-16T08:59:31.204400+0000","last_became_active":"2024-09-16T08:59:05.292409+0000","last_became_peered":"2024-09-16T08:59:05.292409+0000","last_unstale":"2024-09-16T08:59:31.204400+0000","last_undegraded":"2024-09-16T08:59:31.204400+0000","last_fullsized":"2024-09-16T08:59:31.204400+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:22.566068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"61'10","reported_seq":41,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.441392+0000","last_change":"2024-09-16T08:59:01.433702+0000","last_active":"2024-09-16T08:59:30.441392+0000","last_peered":"2024-09-16T08:59:30.441392+0000","last_clean":"2024-09-16T08:59:30.441392+0000","last_became_active":"2024-09-16T08:59:01.433397+0000","last_became_peered":"2024-09-16T08:59:01.433397+0000","last_unstale":"2024-09-16T08:59:30.441392+0000","last_undegraded":"2024-09-16T08:59:30.441392+0000","last_fullsized":"2024-09-16T08:59:30.441392+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:57.100505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440863+0000","last_change":"2024-09-16T08:58:59.264578+0000","last_active":"2024-09-16T08:59:30.440863+0000","last_peered":"2024-09-16T08:59:30.440863+0000","last_clean":"2024-09-16T08:59:30.440863+0000","last_became_active":"2024-09-16T08:58:59.264258+0000","last_became_peered":"2024-09-16T08:58:59.264258+0000","last_unstale":"2024-09-16T08:59:30.440863+0000","last_undegraded":"2024-09-16T08:59:30.440863+0000","last_fullsized":"2024-09-16T08:59:30.440863+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:25.824717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"61'8","reported_seq":39,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:34.318207+0000","last_change":"2024-09-16T08:59:03.281319+0000","last_active":"2024-09-16T08:59:34.318207+0000","last_peered":"2024-09-16T08:59:34.318207+0000","last_clean":"2024-09-16T08:59:34.318207+0000","last_became_active":"2024-09-16T08:59:03.281033+0000","last_became_peered":"2024-09-16T08:59:03.281033+0000","last_unstale":"2024-09-16T08:59:34.318207+0000","last_undegraded":"2024-09-16T08:59:34.318207+0000","last_fullsized":"2024-09-16T08:59:34.318207+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:17:32.662151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204002+0000","last_change":"2024-09-16T08:59:05.291676+0000","last_active":"2024-09-16T08:59:31.204002+0000","last_peered":"2024-09-16T08:59:31.204002+0000","last_clean":"2024-09-16T08:59:31.204002+0000","last_became_active":"2024-09-16T08:59:05.291390+0000","last_became_peered":"2024-09-16T08:59:05.291390+0000","last_unstale":"2024-09-16T08:59:31.204002+0000","last_undegraded":"2024-09-16T08:59:31.204002+0000","last_fullsized":"2024-09-16T08:59:31.204002+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:09:09.160434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"61'11","reported_seq":45,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203914+0000","last_change":"2024-09-16T08:59:01.268104+0000","last_active":"2024-09-16T08:59:31.203914+0000","last_peered":"2024-09-16T08:59:31.203914+0000","last_clean":"2024-09-16T08:59:31.203914+0000","last_became_active":"2024-09-16T08:59:01.267720+0000","last_became_peered":"2024-09-16T08:59:01.267720+0000","last_unstale":"2024-09-16T08:59:31.203914+0000","last_undegraded":"2024-09-16T08:59:31.203914+0000","last_fullsized":"2024-09-16T08:59:31.203914+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:41.383063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204204+0000","last_change":"2024-09-16T08:58:59.266512+0000","last_active":"2024-09-16T08:59:31.204204+0000","last_peered":"2024-09-16T08:59:31.204204+0000","last_clean":"2024-09-16T08:59:31.204204+0000","last_became_active":"2024-09-16T08:58:59.266218+0000","last_became_peered":"2024-09-16T08:58:59.266218+0000","last_unstale":"2024-09-16T08:59:31.204204+0000","last_undegraded":"2024-09-16T08:59:31.204204+0000","last_fullsized":"2024-09-16T08:59:31.204204+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:45.360169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439294+0000","last_change":"2024-09-16T08:59:03.285503+0000","last_active":"2024-09-16T08:59:30.439294+0000","last_peered":"2024-09-16T08:59:30.439294+0000","last_clean":"2024-09-16T08:59:30.439294+0000","last_became_active":"2024-09-16T08:59:03.285244+0000","last_became_peered":"2024-09-16T08:59:03.285244+0000","last_unstale":"2024-09-16T08:59:30.439294+0000","last_undegraded":"2024-09-16T08:59:30.439294+0000","last_fullsized":"2024-09-16T08:59:30.439294+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:55:04.012390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440024+0000","last_change":"2024-09-16T08:59:05.296956+0000","last_active":"2024-09-16T08:59:30.440024+0000","last_peered":"2024-09-16T08:59:30.440024+0000","last_clean":"2024-09-16T08:59:30.440024+0000","last_became_active":"2024-09-16T08:59:05.296628+0000","last_became_peered":"2024-09-16T08:59:05.296628+0000","last_unstale":"2024-09-16T08:59:30.440024+0000","last_undegraded":"2024-09-16T08:59:30.440024+0000","last_fullsized":"2024-09-16T08:59:30.440024+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:18:54.119159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"61'9","reported_seq":42,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439622+0000","last_change":"2024-09-16T08:59:01.267926+0000","last_active":"2024-09-16T08:59:30.439622+0000","last_peered":"2024-09-16T08:59:30.439622+0000","last_clean":"2024-09-16T08:59:30.439622+0000","last_became_active":"2024-09-16T08:59:01.267622+0000","last_became_peered":"2024-09-16T08:59:01.267622+0000","last_unstale":"2024-09-16T08:59:30.439622+0000","last_undegraded":"2024-09-16T08:59:30.439622+0000","last_fullsized":"2024-09-16T08:59:30.439622+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:34:53.973215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440848+0000","last_change":"2024-09-16T08:58:59.267394+0000","last_active":"2024-09-16T08:59:30.440848+0000","last_peered":"2024-09-16T08:59:30.440848+0000","last_clean":"2024-09-16T08:59:30.440848+0000","last_became_active":"2024-09-16T08:58:59.267053+0000","last_became_peered":"2024-09-16T08:58:59.267053+0000","last_unstale":"2024-09-16T08:59:30.440848+0000","last_undegraded":"2024-09-16T08:59:30.440848+0000","last_fullsized":"2024-09-16T08:59:30.440848+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:53.938611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440742+0000","last_change":"2024-09-16T08:59:03.280783+0000","last_active":"2024-09-16T08:59:30.440742+0000","last_peered":"2024-09-16T08:59:30.440742+0000","last_clean":"2024-09-16T08:59:30.440742+0000","last_became_active":"2024-09-16T08:59:03.280492+0000","last_became_peered":"2024-09-16T08:59:03.280492+0000","last_unstale":"2024-09-16T08:59:30.440742+0000","last_undegraded":"2024-09-16T08:59:30.440742+0000","last_fullsized":"2024-09-16T08:59:30.440742+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:48.132611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203029+0000","last_change":"2024-09-16T08:59:05.287326+0000","last_active":"2024-09-16T08:59:31.203029+0000","last_peered":"2024-09-16T08:59:31.203029+0000","last_clean":"2024-09-16T08:59:31.203029+0000","last_became_active":"2024-09-16T08:59:05.287023+0000","last_became_peered":"2024-09-16T08:59:05.287023+0000","last_unstale":"2024-09-16T08:59:31.203029+0000","last_undegraded":"2024-09-16T08:59:31.203029+0000","last_fullsized":"2024-09-16T08:59:31.203029+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:48:38.507456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"61'11","reported_seq":45,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440156+0000","last_change":"2024-09-16T08:59:01.433487+0000","last_active":"2024-09-16T08:59:30.440156+0000","last_peered":"2024-09-16T08:59:30.440156+0000","last_clean":"2024-09-16T08:59:30.440156+0000","last_became_active":"2024-09-16T08:59:01.433074+0000","last_became_peered":"2024-09-16T08:59:01.433074+0000","last_unstale":"2024-09-16T08:59:30.440156+0000","last_undegraded":"2024-09-16T08:59:30.440156+0000","last_fullsized":"2024-09-16T08:59:30.440156+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:05:19.437647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.775821+0000","last_change":"2024-09-16T08:58:59.257127+0000","last_active":"2024-09-16T08:59:30.775821+0000","last_peered":"2024-09-16T08:59:30.775821+0000","last_clean":"2024-09-16T08:59:30.775821+0000","last_became_active":"2024-09-16T08:58:59.256814+0000","last_became_peered":"2024-09-16T08:58:59.256814+0000","last_unstale":"2024-09-16T08:59:30.775821+0000","last_undegraded":"2024-09-16T08:59:30.775821+0000","last_fullsized":"2024-09-16T08:59:30.775821+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:46.796699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439842+0000","last_change":"2024-09-16T08:59:03.284410+0000","last_active":"2024-09-16T08:59:30.439842+0000","last_peered":"2024-09-16T08:59:30.439842+0000","last_clean":"2024-09-16T08:59:30.439842+0000","last_became_active":"2024-09-16T08:59:03.284139+0000","last_became_peered":"2024-09-16T08:59:03.284139+0000","last_unstale":"2024-09-16T08:59:30.439842+0000","last_undegraded":"2024-09-16T08:59:30.439842+0000","last_fullsized":"2024-09-16T08:59:30.439842+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:41:05.484174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440910+0000","last_change":"2024-09-16T08:59:05.294468+0000","last_active":"2024-09-16T08:59:30.440910+0000","last_peered":"2024-09-16T08:59:30.440910+0000","last_clean":"2024-09-16T08:59:30.440910+0000","last_became_active":"2024-09-16T08:59:05.294148+0000","last_became_peered":"2024-09-16T08:59:05.294148+0000","last_unstale":"2024-09-16T08:59:30.440910+0000","last_undegraded":"2024-09-16T08:59:30.440910+0000","last_fullsized":"2024-09-16T08:59:30.440910+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:52:12.649060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"61'4","reported_seq":32,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440111+0000","last_change":"2024-09-16T08:59:01.270548+0000","last_active":"2024-09-16T08:59:30.440111+0000","last_peered":"2024-09-16T08:59:30.440111+0000","last_clean":"2024-09-16T08:59:30.440111+0000","last_became_active":"2024-09-16T08:59:01.270274+0000","last_became_peered":"2024-09-16T08:59:01.270274+0000","last_unstale":"2024-09-16T08:59:30.440111+0000","last_undegraded":"2024-09-16T08:59:30.440111+0000","last_fullsized":"2024-09-16T08:59:30.440111+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:18.014009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204734+0000","last_change":"2024-09-16T08:58:59.257328+0000","last_active":"2024-09-16T08:59:31.204734+0000","last_peered":"2024-09-16T08:59:31.204734+0000","last_clean":"2024-09-16T08:59:31.204734+0000","last_became_active":"2024-09-16T08:58:59.257061+0000","last_became_peered":"2024-09-16T08:58:59.257061+0000","last_unstale":"2024-09-16T08:59:31.204734+0000","last_undegraded":"2024-09-16T08:59:31.204734+0000","last_fullsized":"2024-09-16T08:59:31.204734+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:53:18.183635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.202825+0000","last_change":"2024-09-16T08:59:03.283069+0000","last_active":"2024-09-16T08:59:31.202825+0000","last_peered":"2024-09-16T08:59:31.202825+0000","last_clean":"2024-09-16T08:59:31.202825+0000","last_became_active":"2024-09-16T08:59:03.282791+0000","last_became_peered":"2024-09-16T08:59:03.282791+0000","last_unstale":"2024-09-16T08:59:31.202825+0000","last_undegraded":"2024-09-16T08:59:31.202825+0000","last_fullsized":"2024-09-16T08:59:31.202825+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:23:43.786012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"61'1","reported_seq":19,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440396+0000","last_change":"2024-09-16T08:59:05.290227+0000","last_active":"2024-09-16T08:59:30.440396+0000","last_peered":"2024-09-16T08:59:30.440396+0000","last_clean":"2024-09-16T08:59:30.440396+0000","last_became_active":"2024-09-16T08:59:05.289940+0000","last_became_peered":"2024-09-16T08:59:05.289940+0000","last_unstale":"2024-09-16T08:59:30.440396+0000","last_undegraded":"2024-09-16T08:59:30.440396+0000","last_fullsized":"2024-09-16T08:59:30.440396+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:18:13.275619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.439478+0000","last_change":"2024-09-16T08:59:05.289802+0000","last_active":"2024-09-16T08:59:30.439478+0000","last_peered":"2024-09-16T08:59:30.439478+0000","last_clean":"2024-09-16T08:59:30.439478+0000","last_became_active":"2024-09-16T08:59:05.289414+0000","last_became_peered":"2024-09-16T08:59:05.289414+0000","last_unstale":"2024-09-16T08:59:30.439478+0000","last_undegraded":"2024-09-16T08:59:30.439478+0000","last_fullsized":"2024-09-16T08:59:30.439478+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:04.268176+0000","last_clean_scrub_stamp":"2024-09-16T08:59:04.268176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:14.377503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"54'1","reported_seq":36,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:30.440438+0000","last_change":"2024-09-16T08:58:59.255988+0000","last_active":"2024-09-16T08:59:30.440438+0000","last_peered":"2024-09-16T08:59:30.440438+0000","last_clean":"2024-09-16T08:59:30.440438+0000","last_became_active":"2024-09-16T08:58:59.255595+0000","last_became_peered":"2024-09-16T08:58:59.255595+0000","last_unstale":"2024-09-16T08:59:30.440438+0000","last_undegraded":"2024-09-16T08:59:30.440438+0000","last_fullsized":"2024-09-16T08:59:30.440438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:58:58.234927+0000","last_clean_scrub_stamp":"2024-09-16T08:58:58.234927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:03.865357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"61'11","reported_seq":45,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.203763+0000","last_change":"2024-09-16T08:59:01.271803+0000","last_active":"2024-09-16T08:59:31.203763+0000","last_peered":"2024-09-16T08:59:31.203763+0000","last_clean":"2024-09-16T08:59:31.203763+0000","last_became_active":"2024-09-16T08:59:01.271468+0000","last_became_peered":"2024-09-16T08:59:01.271468+0000","last_unstale":"2024-09-16T08:59:31.203763+0000","last_undegraded":"2024-09-16T08:59:31.203763+0000","last_fullsized":"2024-09-16T08:59:31.203763+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:00.245943+0000","last_clean_scrub_stamp":"2024-09-16T08:59:00.245943+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:31.826849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":22,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T08:59:31.204157+0000","last_change":"2024-09-16T08:59:03.283781+0000","last_active":"2024-09-16T08:59:31.204157+0000","last_peered":"2024-09-16T08:59:31.204157+0000","last_clean":"2024-09-16T08:59:31.204157+0000","last_became_active":"2024-09-16T08:59:03.282976+0000","last_became_peered":"2024-09-16T08:59:03.282976+0000","last_unstale":"2024-09-16T08:59:31.204157+0000","last_undegraded":"2024-09-16T08:59:31.204157+0000","last_fullsized":"2024-09-16T08:59:31.204157+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T08:59:02.262812+0000","last_clean_scrub_stamp":"2024-09-16T08:59:02.262812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:15.809446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":32,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397518,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32452,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712700,"statfs":{"total":95995035648,"available":95961804800,"internally_reserved":0,"allocated":1298432,"data_stored":877431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593747,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32428,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712724,"statfs":{"total":95995035648,"available":95961829376,"internally_reserved":0,"allocated":1273856,"data_stored":876333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789975,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":286324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":716800,"data_stored":287576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":720896,"data_stored":287292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":287119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":753664,"data_stored":289179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28300,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716852,"statfs":{"total":95995035648,"available":95966056448,"internally_reserved":0,"allocated":1302528,"data_stored":877563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T08:59:46.251 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T08:59:46.251 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T08:59:46.252 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-16T08:59:46.252 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705 shell --fsid f774c998-7408-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-16T08:59:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:46 smithi111 ceph-mon[29803]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-16T08:59:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:46 smithi111 ceph-mon[29803]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T08:59:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T08:59:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:46 smithi111 ceph-mon[40056]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-16T08:59:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:46 smithi111 ceph-mon[40056]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T08:59:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T08:59:46.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:46 smithi114 ceph-mon[33315]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-16T08:59:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:46 smithi114 ceph-mon[33315]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T08:59:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T08:59:47.124 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T08:59:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.667+0000 7f7440fd2640 1 -- 172.21.15.111:0/2522382743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f743c109150 msgr2=0x7f743c10f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.667+0000 7f7440fd2640 1 --2- 172.21.15.111:0/2522382743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f743c109150 0x7f743c10f950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f742c009900 tx=0x7f742c02a410 comp rx=0 tx=0).stop 2024-09-16T08:59:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.668+0000 7f7440fd2640 1 -- 172.21.15.111:0/2522382743 shutdown_connections 2024-09-16T08:59:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.668+0000 7f7440fd2640 1 --2- 172.21.15.111:0/2522382743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f743c109150 0x7f743c10f950 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.668+0000 7f7440fd2640 1 --2- 172.21.15.111:0/2522382743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f743c1087d0 0x7f743c108c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.668+0000 7f7440fd2640 1 --2- 172.21.15.111:0/2522382743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c102dc0 0x7f743c1031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.668+0000 7f7440fd2640 1 -- 172.21.15.111:0/2522382743 >> 172.21.15.111:0/2522382743 conn(0x7f743c0fe810 msgr2=0x7f743c100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.668+0000 7f7440fd2640 1 -- 172.21.15.111:0/2522382743 shutdown_connections 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.668+0000 7f7440fd2640 1 -- 172.21.15.111:0/2522382743 wait complete. 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.668+0000 7f7440fd2640 1 Processor -- start 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.668+0000 7f7440fd2640 1 -- start start 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f7440fd2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f743c102dc0 0x7f743c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f7440fd2640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f743c1087d0 0x7f743c198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f7440fd2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c109150 0x7f743c19e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f7440fd2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f743c1122e0 con 0x7f743c102dc0 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f7440fd2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f743c112160 con 0x7f743c109150 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f7440fd2640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f743c112460 con 0x7f743c1087d0 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f7439d9b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f743c1087d0 0x7f743c198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743a59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f743c102dc0 0x7f743c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743a59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f743c102dc0 0x7f743c198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33194/0 (socket says 172.21.15.111:33194) 2024-09-16T08:59:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743a59c640 1 -- 172.21.15.111:0/1769097058 learned_addr learned my addr 172.21.15.111:0/1769097058 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743ad9d640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c109150 0x7f743c19e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743a59c640 1 -- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f743c1087d0 msgr2=0x7f743c198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743a59c640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f743c1087d0 0x7f743c198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743a59c640 1 -- 172.21.15.111:0/1769097058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c109150 msgr2=0x7f743c19e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743a59c640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c109150 0x7f743c19e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743a59c640 1 -- 172.21.15.111:0/1769097058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743c19e840 con 0x7f743c102dc0 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f743a59c640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f743c102dc0 0x7f743c198790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f743400d770 tx=0x7f743400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f742b7fe640 1 -- 172.21.15.111:0/1769097058 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434017070 con 0x7f743c102dc0 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f743c19eb30 con 0x7f743c102dc0 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f742b7fe640 1 -- 172.21.15.111:0/1769097058 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f7434005470 con 0x7f743c102dc0 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f742b7fe640 1 -- 172.21.15.111:0/1769097058 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434004b40 con 0x7f743c102dc0 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.669+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f743c1a51f0 con 0x7f743c102dc0 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.670+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f743c1031a0 con 0x7f743c102dc0 2024-09-16T08:59:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.671+0000 7f742b7fe640 1 -- 172.21.15.111:0/1769097058 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f7434004ce0 con 0x7f743c102dc0 2024-09-16T08:59:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.671+0000 7f742b7fe640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f742407ae10 0x7f742407d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T08:59:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.672+0000 7f7439d9b640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f742407ae10 0x7f742407d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T08:59:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.672+0000 7f742b7fe640 1 -- 172.21.15.111:0/1769097058 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7421+0+0 (secure 0 0 0) 0x7f743409dec0 con 0x7f743c102dc0 2024-09-16T08:59:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.672+0000 7f7439d9b640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f742407ae10 0x7f742407d2d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f74300099a0 tx=0x7f7430009500 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T08:59:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.676+0000 7f742b7fe640 1 -- 172.21.15.111:0/1769097058 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f7434069170 con 0x7f743c102dc0 2024-09-16T08:59:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.893+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f743c10af30 con 0x7f743c102dc0 2024-09-16T08:59:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.894+0000 7f742b7fe640 1 -- 172.21.15.111:0/1769097058 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f743406e020 con 0x7f743c102dc0 2024-09-16T08:59:47.895 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-09-16T08:59:47.895 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f742407ae10 msgr2=0x7f742407d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f742407ae10 0x7f742407d2d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f74300099a0 tx=0x7f7430009500 comp rx=0 tx=0).stop 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f743c102dc0 msgr2=0x7f743c198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f743c102dc0 0x7f743c198790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f743400d770 tx=0x7f743400dc30 comp rx=0 tx=0).stop 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 shutdown_connections 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f742407ae10 0x7f742407d2d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743c109150 0x7f743c19e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f743c1087d0 0x7f743c198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 --2- 172.21.15.111:0/1769097058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f743c102dc0 0x7f743c198790 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T08:59:47.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 >> 172.21.15.111:0/1769097058 conn(0x7f743c0fe810 msgr2=0x7f743c100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T08:59:47.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 shutdown_connections 2024-09-16T08:59:47.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-09-16T08:59:47.896+0000 7f7440fd2640 1 -- 172.21.15.111:0/1769097058 wait complete. 2024-09-16T08:59:48.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:48 smithi111 ceph-mon[29803]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-16T08:59:48.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1769097058' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-16T08:59:48.515 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:48 smithi111 ceph-mon[40056]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-16T08:59:48.515 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1769097058' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-16T08:59:48.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:48 smithi114 ceph-mon[33315]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-16T08:59:48.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1769097058' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-16T08:59:50.115 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-16T08:59:50.115 INFO:tasks.cephadm:Setup complete, yielding 2024-09-16T08:59:50.115 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T08:59:50.129 INFO:tasks.workunit:Pulling workunits from ref 8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:59:50.130 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T08:59:50.130 DEBUG:teuthology.orchestra.run.smithi111:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T08:59:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T08:59:50.148 INFO:teuthology.orchestra.run.smithi111.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T08:59:50.149 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T08:59:50.206 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T08:59:50.206 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T08:59:50.263 INFO:tasks.workunit:timeout=3h 2024-09-16T08:59:50.263 INFO:tasks.workunit:cleanup=True 2024-09-16T08:59:50.263 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8293d73f8690540e843a81caec373f9cc29cf705 2024-09-16T08:59:50.320 INFO:tasks.workunit.client.0.smithi111.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T08:59:50.425 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:50 smithi111 ceph-mon[40056]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-16T08:59:50.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:50 smithi111 ceph-mon[29803]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-16T08:59:50.466 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:50 smithi114 ceph-mon[33315]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-16T08:59:51.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:51 smithi111 systemd[1]: Starting Ceph alertmanager.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T08:59:51.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 08:59:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T08:59:52.082 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:51 smithi111 podman[74558]: 2024-09-16 08:59:51.791462578 +0000 UTC m=+0.111764434 volume create 0804d2f8221f110dedb704b2e6bc28d0f80916a8dbb2d4f4a445092a4b996b7b 2024-09-16T08:59:52.082 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:51 smithi111 podman[74558]: 2024-09-16 08:59:51.69536793 +0000 UTC m=+0.015669792 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-16T08:59:52.082 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:51 smithi111 podman[74558]: 2024-09-16 08:59:51.875203064 +0000 UTC m=+0.195504962 container create 13e7816bc96cd7c4fcdb3401ff2eee1e75ac1b2b651b00e3f5f6127ce4b543d6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T08:59:52.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:52 smithi114 ceph-mon[33315]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T08:59:52.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-mon[29803]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T08:59:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-mon[40056]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T08:59:52.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 podman[74558]: 2024-09-16 08:59:52.186645066 +0000 UTC m=+0.506946923 container init 13e7816bc96cd7c4fcdb3401ff2eee1e75ac1b2b651b00e3f5f6127ce4b543d6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T08:59:52.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 podman[74558]: 2024-09-16 08:59:52.191560172 +0000 UTC m=+0.511862028 container start 13e7816bc96cd7c4fcdb3401ff2eee1e75ac1b2b651b00e3f5f6127ce4b543d6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T08:59:52.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T08:59:52.204Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-16T08:59:52.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T08:59:52.204Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-16T08:59:52.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T08:59:52.205Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.111 port=9094 2024-09-16T08:59:52.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T08:59:52.205Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-16T08:59:52.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T08:59:52.227Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-16T08:59:52.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T08:59:52.227Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-16T08:59:52.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T08:59:52.229Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-16T08:59:52.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T08:59:52.229Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-16T08:59:52.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 bash[74558]: 13e7816bc96cd7c4fcdb3401ff2eee1e75ac1b2b651b00e3f5f6127ce4b543d6 2024-09-16T08:59:52.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:52 smithi111 systemd[1]: Started Ceph alertmanager.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T08:59:53.329 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T08:59:53.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-16T08:59:53.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.330 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:08:59:53] ENGINE Bus STOPPING 2024-09-16T08:59:53.331 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:08:59:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T08:59:53.332 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:08:59:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T08:59:53.332 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:08:59:53] ENGINE Bus STOPPED 2024-09-16T08:59:53.332 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:08:59:53] ENGINE Bus STARTING 2024-09-16T08:59:53.332 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:08:59:53] ENGINE Serving on http://:::9283 2024-09-16T08:59:53.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T08:59:53.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-16T08:59:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:53.675 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 08:59:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:08:59:53] ENGINE Bus STARTED 2024-09-16T08:59:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:54 smithi114 ceph-mon[33315]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-16T08:59:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:54 smithi114 ceph-mon[33315]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T08:59:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:54 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-16T08:59:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:54 smithi114 ceph-mon[33315]: Deploying daemon grafana.a on smithi114 2024-09-16T08:59:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:54 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T08:59:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:54 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T08:59:54.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T08:59:54.206Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000098426s 2024-09-16T08:59:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[29803]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-16T08:59:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[29803]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T08:59:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-16T08:59:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[29803]: Deploying daemon grafana.a on smithi114 2024-09-16T08:59:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T08:59:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T08:59:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[40056]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-16T08:59:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[40056]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T08:59:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-16T08:59:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[40056]: Deploying daemon grafana.a on smithi114 2024-09-16T08:59:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T08:59:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:54 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T08:59:56.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:56 smithi114 ceph-mon[33315]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T08:59:56.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:56 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:56 smithi111 ceph-mon[29803]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T08:59:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:56 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:56 smithi111 ceph-mon[40056]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T08:59:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:56 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T08:59:58.490 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 08:59:58 smithi114 ceph-mon[33315]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T08:59:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 08:59:58 smithi111 ceph-mon[29803]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T08:59:58.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 08:59:58 smithi111 ceph-mon[40056]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:00.478 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:00 smithi111 ceph-mon[29803]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:00.478 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:00 smithi111 ceph-mon[29803]: overall HEALTH_OK 2024-09-16T09:00:00.478 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:00 smithi111 ceph-mon[40056]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:00.479 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:00 smithi111 ceph-mon[40056]: overall HEALTH_OK 2024-09-16T09:00:00.481 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:00 smithi114 ceph-mon[33315]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:00.481 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:00 smithi114 ceph-mon[33315]: overall HEALTH_OK 2024-09-16T09:00:01.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:01.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:01.750 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:00:01 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:00:02.361 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:02 smithi114 ceph-mon[33315]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:02 smithi111 ceph-mon[29803]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:02 smithi111 ceph-mon[40056]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:02.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:02 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T09:00:02.209Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002997027s 2024-09-16T09:00:03.045 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 systemd[1]: Starting Ceph grafana.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:00:03.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:03.414 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 podman[63126]: 2024-09-16 09:00:03.157171427 +0000 UTC m=+0.016570418 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-09-16T09:00:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:03 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:03 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:00:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:00:03.843 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 podman[63126]: 2024-09-16 09:00:03.533167964 +0000 UTC m=+0.392566944 container create b180e6da2fe4041b6cf7974e65a876325f2c0d82ae6109beba933b31baed19f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T09:00:03.844 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 podman[63126]: 2024-09-16 09:00:03.752265558 +0000 UTC m=+0.611664544 container init b180e6da2fe4041b6cf7974e65a876325f2c0d82ae6109beba933b31baed19f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T09:00:03.844 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 podman[63126]: 2024-09-16 09:00:03.757058044 +0000 UTC m=+0.616457028 container start b180e6da2fe4041b6cf7974e65a876325f2c0d82ae6109beba933b31baed19f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T09:00:04.094 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 bash[63126]: b180e6da2fe4041b6cf7974e65a876325f2c0d82ae6109beba933b31baed19f8 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 systemd[1]: Started Ceph grafana.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877704319Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-09-16T09:00:03Z 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877929393Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877937494Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877940632Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877945274Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877947659Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877950067Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877952614Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877955503Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-16T09:00:04.095 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877958047Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877960394Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877962833Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877967506Z level=info msg=Target target=[all] 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877976916Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877979494Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877981838Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877989741Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877992412Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=settings t=2024-09-16T09:00:03.877995052Z level=info msg="App mode production" 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore t=2024-09-16T09:00:03.878255167Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore t=2024-09-16T09:00:03.878267366Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-16T09:00:04.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.878778338Z level=info msg="Starting DB migrations" 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.879653652Z level=info msg="Executing migration" id="create migration_log table" 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.880427433Z level=info msg="Migration successfully executed" id="create migration_log table" duration=794.271µs 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.88091933Z level=info msg="Executing migration" id="create user table" 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.881445552Z level=info msg="Migration successfully executed" id="create user table" duration=526.084µs 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.881882062Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.882403279Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=521.414µs 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.882819064Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.883270415Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=451.299µs 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.883664394Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.884141407Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=483.778µs 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.884682243Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.885140931Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=454.286µs 2024-09-16T09:00:04.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.885501493Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-16T09:00:04.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.88709535Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=1.593476ms 2024-09-16T09:00:04.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.887548232Z level=info msg="Executing migration" id="create user table v2" 2024-09-16T09:00:04.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.888167117Z level=info msg="Migration successfully executed" id="create user table v2" duration=615.306µs 2024-09-16T09:00:04.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.888546352Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-16T09:00:04.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.889019842Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=473.539µs 2024-09-16T09:00:04.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.889372041Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-16T09:00:04.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.889869163Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=496.72µs 2024-09-16T09:00:04.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.890321771Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-16T09:00:04.100 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.890794404Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=469.498µs 2024-09-16T09:00:04.100 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.891135873Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-16T09:00:04.100 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.891510906Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=371.932µs 2024-09-16T09:00:04.100 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.891854391Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-16T09:00:04.100 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.892559906Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=705.269µs 2024-09-16T09:00:04.100 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.892905185Z level=info msg="Executing migration" id="Update user table charset" 2024-09-16T09:00:04.100 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.892926266Z level=info msg="Migration successfully executed" id="Update user table charset" duration=21.632µs 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.893313951Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.894031666Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=717.169µs 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.894367755Z level=info msg="Executing migration" id="Add missing user data" 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.894503964Z level=info msg="Migration successfully executed" id="Add missing user data" duration=134.721µs 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.894909058Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.89567764Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=767.371µs 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.89603503Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.896506042Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=468.647µs 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.897225413Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.897957253Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=731.473µs 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.898296146Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-16T09:00:04.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.903556305Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=5.259536ms 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.903883293Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.904595703Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=712.284µs 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.904935031Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.905070382Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=134.836µs 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.905476392Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.905937526Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=461.165µs 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.906359412Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.90695268Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=591.554µs 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.907330919Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.9078128Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=481.68µs 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.908184432Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.908661541Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=476.807µs 2024-09-16T09:00:04.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.909026075Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.909515567Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=489.322µs 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.90989539Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.910380445Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=484.851µs 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.910741872Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.910761691Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=20.972µs 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.911209791Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.911690552Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=478.759µs 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.912057646Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.912530344Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=473.158µs 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.912899341Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.913368488Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=469.431µs 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.913731171Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-16T09:00:04.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.914178009Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=446.865µs 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.914528709Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.916500791Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=1.971794ms 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.916888543Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.91744282Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=554.12µs 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.917805533Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.918285761Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=480.053µs 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.918638901Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.919112827Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=473.709µs 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.919463524Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.919952958Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=489.227µs 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.920318689Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.920798628Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=479.802µs 2024-09-16T09:00:04.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.921206996Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.921486572Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=279.334µs 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.921839195Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.922228039Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=388.756µs 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.922584178Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.922862856Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=278.599µs 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.923264026Z level=info msg="Executing migration" id="create star table" 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.923728386Z level=info msg="Migration successfully executed" id="create star table" duration=467.22µs 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.924074278Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.924575889Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=501.321µs 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.924964079Z level=info msg="Executing migration" id="create org table v1" 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.925469132Z level=info msg="Migration successfully executed" id="create org table v1" duration=503.469µs 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.925837488Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-16T09:00:04.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.926323039Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=485.404µs 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.926733535Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.927186063Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=452.264µs 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.927603956Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.928161631Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=558.406µs 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.928532297Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.929055405Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=522.81µs 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.929448411Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.929956556Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=507.917µs 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.930313453Z level=info msg="Executing migration" id="Update org table charset" 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.930328217Z level=info msg="Migration successfully executed" id="Update org table charset" duration=15.46µs 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.9311689Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.931186678Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=18.817µs 2024-09-16T09:00:04.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.932163474Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.932380872Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=221.285µs 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.932769957Z level=info msg="Executing migration" id="create dashboard table" 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.933279287Z level=info msg="Migration successfully executed" id="create dashboard table" duration=509.635µs 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.933651937Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.934176558Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=524.33µs 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.934542007Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.935122291Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=581.358µs 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.935509283Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.936018593Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=509.392µs 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.936416913Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.936934613Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=517.545µs 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.937376416Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.937844436Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=467.849µs 2024-09-16T09:00:04.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.93821108Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.941559456Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.348019ms 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.94193877Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.942435877Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=496.89µs 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.942809011Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.943279362Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=470.15µs 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.945866777Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.946368157Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=501.329µs 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.94869632Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.94893671Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=240.339µs 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.95141906Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.952044395Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=625.223µs 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.95240849Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-16T09:00:04.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.952446856Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=38.753µs 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.95524628Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.956344179Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.10009ms 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.958165667Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.959295763Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.130299ms 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.960718479Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.961807827Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.08954ms 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.962186898Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.962700527Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=513.706µs 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.964803259Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.965875705Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.072332ms 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.96773156Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.968241666Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=509.942µs 2024-09-16T09:00:04.109 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.970437682Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.970957865Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=521.788µs 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.971323218Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.971341875Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=19.618µs 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.974015865Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.974035565Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=20.829µs 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.974470443Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.975624609Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.154446ms 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.977391511Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.978515643Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.123922ms 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.97890049Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.980079992Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.179298ms 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.981856528Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.983029168Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.172572ms 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.984421519Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-16T09:00:04.110 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.984564996Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=143.55µs 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.987045131Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.987547692Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=500.578µs 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.98792297Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.988411364Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=488.067µs 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.990718579Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.990735984Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=18.467µs 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.99339209Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.993928691Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=536.809µs 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.996070674Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.996533034Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=461.95µs 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:03 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:03.996928198Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.000032979Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=3.120071ms 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.000409535Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-16T09:00:04.111 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.000877843Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=468.122µs 2024-09-16T09:00:04.112 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.003108686Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-16T09:00:04.112 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.003599216Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=489.773µs 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.006057324Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.006600577Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=543.166µs 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.006984703Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.007218401Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=233.42µs 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.009896471Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.010382472Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=488.35µs 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.012496436Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.013756757Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.259822ms 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.015271437Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-16T09:00:04.113 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.015922999Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=654.097µs 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.016286453Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.016412661Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=126.263µs 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.019029638Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.019173952Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=144.175µs 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.021721211Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.022226281Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=504.865µs 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.022649923Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.023908451Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.259901ms 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.024310565Z level=info msg="Executing migration" id="create data_source table" 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.024957393Z level=info msg="Migration successfully executed" id="create data_source table" duration=646.618µs 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.025361196Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.025872179Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=510.62µs 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.02626615Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-16T09:00:04.114 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.026777123Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=511.447µs 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.027173617Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.027664807Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=489.3µs 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.028019723Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.028500465Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=480.583µs 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.030937418Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.034333589Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.398439ms 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.034752151Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.035354666Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=602.253µs 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.037618593Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.03819928Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=580.439µs 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.038571627Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.039112645Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=540.711µs 2024-09-16T09:00:04.115 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.041385973Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.041815724Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=427.999µs 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.044214447Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.045752402Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.540229ms 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.046986696Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.048425051Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.438143ms 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.048811929Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.048827393Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=16.195µs 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.051536739Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.051712964Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=175.963µs 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.054252264Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.055695749Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.443162ms 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.057182539Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.057434572Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=254.57µs 2024-09-16T09:00:04.116 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.057863095Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.058001958Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=138.84µs 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.060400874Z level=info msg="Executing migration" id="Add uid column" 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.061818182Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.416994ms 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.063171306Z level=info msg="Executing migration" id="Update uid value" 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.063335188Z level=info msg="Migration successfully executed" id="Update uid value" duration=163.785µs 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.065869735Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.06641681Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=546.969µs 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.066852217Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.067359145Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=506.644µs 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.069447708Z level=info msg="Executing migration" id="create api_key table" 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.07005433Z level=info msg="Migration successfully executed" id="create api_key table" duration=604.516µs 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.072035056Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-16T09:00:04.117 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.072581056Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=545.836µs 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.074756827Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.075275365Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=518.45µs 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.075689707Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.076246521Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=555.065µs 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.078376133Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.078918587Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=542.44µs 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.081170358Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.081698642Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=528.178µs 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.084064227Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.084692905Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=630.851µs 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.086718187Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.090511206Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=3.79277ms 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.090901641Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-16T09:00:04.118 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.091406272Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=504.352µs 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.091763264Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.092271715Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=508.361µs 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.092636276Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.09315231Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=516.142µs 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.09354695Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.094078725Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=531.605µs 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.094491838Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.094748402Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=256.51µs 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.095265419Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.095692992Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=427.494µs 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.096088378Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.096104357Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=16.683µs 2024-09-16T09:00:04.119 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.096877876Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-16T09:00:04.120 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.098440373Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.562176ms 2024-09-16T09:00:04.120 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.098813889Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-16T09:00:04.120 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.100358728Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=1.544516ms 2024-09-16T09:00:04.120 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.100757354Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-16T09:00:04.120 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.100881645Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=124.389µs 2024-09-16T09:00:04.120 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.101514343Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-16T09:00:04.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-mon[33315]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:04.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.346 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.10308473Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.568402ms 2024-09-16T09:00:04.346 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.103447382Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-16T09:00:04.346 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.104959005Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.511233ms 2024-09-16T09:00:04.346 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.1053244Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-16T09:00:04.346 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.105863189Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=538.655µs 2024-09-16T09:00:04.346 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.10621853Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-16T09:00:04.346 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.106610375Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=391.731µs 2024-09-16T09:00:04.346 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.107077551Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-16T09:00:04.346 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.107663666Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=585.81µs 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.108025873Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.108563722Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=537.698µs 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.108937972Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.109509802Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=571.596µs 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.110639588Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.111626218Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=996.423µs 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.112082835Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.112143686Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=61.198µs 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.112905759Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.112923519Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=19.422µs 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.113752077Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.115383783Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.631302ms 2024-09-16T09:00:04.347 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.115730746Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.117321836Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=1.59085ms 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.117744809Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.117783264Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=39.02µs 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.118588951Z level=info msg="Executing migration" id="create quota table v1" 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.119060125Z level=info msg="Migration successfully executed" id="create quota table v1" duration=470.927µs 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.119415717Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.119911084Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=495.044µs 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.120258762Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.120273612Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=15.503µs 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.121150842Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.12164696Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=495.907µs 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.121987958Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-16T09:00:04.348 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.122490239Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=501.983µs 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.122831321Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.124605248Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=1.773426ms 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.124948082Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.124973677Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=19.603µs 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.125962109Z level=info msg="Executing migration" id="create session table" 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.126589202Z level=info msg="Migration successfully executed" id="create session table" duration=629.522µs 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.126992119Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.12707894Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=87.129µs 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.127822181Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.127891013Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=68.878µs 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.128870581Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.129437027Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=568.275µs 2024-09-16T09:00:04.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.129797223Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-16T09:00:04.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.130284064Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=486.568µs 2024-09-16T09:00:04.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.13066914Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-16T09:00:04.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.130689253Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=20.32µs 2024-09-16T09:00:04.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.133391944Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-16T09:00:04.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.133409663Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=17.978µs 2024-09-16T09:00:04.351 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.136235576Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.138072399Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=1.835039ms 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.139164932Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.140986705Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=1.821356ms 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.141873532Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.141944182Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=69.717µs 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.144473721Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.144558096Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=84.387µs 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.147232504Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.147790027Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=557.766µs 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.149864867Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.149880177Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=16.043µs 2024-09-16T09:00:04.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.152752838Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.154608686Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=1.856809ms 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.155539446Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.155771691Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=234.923µs 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.158370807Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.160212381Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=1.841387ms 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.161431582Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.163239884Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.812371ms 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.164212169Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.16431113Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=102.025µs 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.167099011Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.167831326Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=734.823µs 2024-09-16T09:00:04.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.169858373Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.170470118Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=614.015µs 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.172662002Z level=info msg="Executing migration" id="create alert table v1" 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.173372365Z level=info msg="Migration successfully executed" id="create alert table v1" duration=710.012µs 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.17550773Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.176219279Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=713.784µs 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.178211613Z level=info msg="Executing migration" id="add index alert state" 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.178856573Z level=info msg="Migration successfully executed" id="add index alert state" duration=646.819µs 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.180747724Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.181289228Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=542.55µs 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.18340683Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.183883756Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=476.5µs 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.186057899Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-16T09:00:04.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.186639635Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=582.637µs 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.188860329Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.18937703Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=516.556µs 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.19167329Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.197392392Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=5.718764ms 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.19774931Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.198217976Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=468.527µs 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.200336454Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.200900762Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=563.91µs 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.203292208Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.203573738Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=286.377µs 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.205906508Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.206281204Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=374.408µs 2024-09-16T09:00:04.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.208737189Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.209227752Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=490.387µs 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.21144015Z level=info msg="Executing migration" id="Add column is_default" 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.213479245Z level=info msg="Migration successfully executed" id="Add column is_default" duration=2.037191ms 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.213972337Z level=info msg="Executing migration" id="Add column frequency" 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.216240377Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.267722ms 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.216754869Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.2190611Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.30602ms 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.219398443Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.22147594Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.077242ms 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.222169708Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.222716005Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=546.055µs 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.224852159Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-16T09:00:04.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.224871376Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=19.369µs 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.227591012Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.227610582Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=19.674µs 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.230206234Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.230710147Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=503.518µs 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.231067664Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.231594495Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=526.607µs 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.231961969Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.232487878Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=525.773µs 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.23286494Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.233376021Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=510.77µs 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.233731529Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.234261909Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=530.139µs 2024-09-16T09:00:04.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.23459988Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.236758804Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.158547ms 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.23711941Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.239333814Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.2151ms 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.239705405Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.2398303Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=124.052µs 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.240555573Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.241095214Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=539.386µs 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.241489725Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.242021072Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=531.244µs 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.242359264Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.244469635Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.110035ms 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.244833678Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-16T09:00:04.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.244882044Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=48.558µs 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.245683973Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.246214729Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=530.514µs 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.246567647Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.247179821Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=611.94µs 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.247550904Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.247616302Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=65.494µs 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.248392373Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.248991059Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=599.997µs 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.249356831Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.249888826Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=531.669µs 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.25026942Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.250813055Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=543.236µs 2024-09-16T09:00:04.359 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.251199121Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.251730171Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=530.773µs 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.252095422Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.252703126Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=606.14µs 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.253076995Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.253704886Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=627.403µs 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.254065122Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.254084675Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=18.935µs 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.254864009Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.257326968Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.462682ms 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.257703352Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.258227266Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=523.882µs 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.258587802Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-16T09:00:04.360 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.260860468Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.272238ms 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.261312864Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.261768676Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=455.629µs 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.264113127Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.264670985Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=557.656µs 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.266737778Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.267275062Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=537.37µs 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.269719535Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.276343596Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=6.626046ms 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.276751668Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.277245502Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=494.153µs 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.279483793Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.280053901Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=574.724µs 2024-09-16T09:00:04.361 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.282156673Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.282362492Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=205.745µs 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.285037202Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.285428571Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=391.506µs 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.287782165Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.288037707Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=257.995µs 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.290285613Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.292579781Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.29393ms 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.293010842Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.295312064Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.301086ms 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.295695404Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.296237381Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=541.681µs 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.29858675Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-16T09:00:04.362 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.299226598Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=641.938µs 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.301184621Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.301354474Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=169.934µs 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.303753975Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.306108108Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.353665ms 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.30648995Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.3070483Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=557.15µs 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.30941134Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.30957628Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=164.616µs 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.312006979Z level=info msg="Executing migration" id="Move region to single row" 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.312335237Z level=info msg="Migration successfully executed" id="Move region to single row" duration=329.599µs 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.312763889Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.313349664Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=586.126µs 2024-09-16T09:00:04.363 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.313700698Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.314244004Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=543.105µs 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.314588407Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.315205212Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=616.377µs 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.315544298Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.316099867Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=556.529µs 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.316433039Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.316986279Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=553.072µs 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.31736459Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.317908949Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=544.71µs 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.318303481Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.318346438Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=43.8µs 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.319102637Z level=info msg="Executing migration" id="create test_data table" 2024-09-16T09:00:04.364 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.319661687Z level=info msg="Migration successfully executed" id="create test_data table" duration=558.519µs 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.320054719Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.320573483Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=518.926µs 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.32093872Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.321472908Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=534.486µs 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.321822575Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.322396845Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=573.268µs 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.322749871Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.322866601Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=116.697µs 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.323642747Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.323888743Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=245.932µs 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.324420099Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-16T09:00:04.365 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.324459675Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=40.249µs 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.325227074Z level=info msg="Executing migration" id="create team table" 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.32571001Z level=info msg="Migration successfully executed" id="create team table" duration=482.682µs 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.326076567Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.326695719Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=618.87µs 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.32704782Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.327585259Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=536.775µs 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.327937357Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.330424744Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.487229ms 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.330783223Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.330892204Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=109.196µs 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.331642339Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.332218102Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=573.345µs 2024-09-16T09:00:04.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.332586084Z level=info msg="Executing migration" id="create team member table" 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.333090053Z level=info msg="Migration successfully executed" id="create team member table" duration=503.815µs 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.333463096Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.334013878Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=550.388µs 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.334352978Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.334899248Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=545.941µs 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.335237438Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.335775108Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=532.345µs 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.336122199Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.33882309Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=2.700495ms 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.339170743Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.341988573Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=2.818291ms 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.342343103Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.344937245Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.592885ms 2024-09-16T09:00:04.367 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.345378313Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-16T09:00:04.368 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.346151576Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=772.792µs 2024-09-16T09:00:04.368 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.34655525Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-16T09:00:04.368 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.347274297Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=719.329µs 2024-09-16T09:00:04.368 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.347678875Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-16T09:00:04.368 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.348456978Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=777.693µs 2024-09-16T09:00:04.368 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.348867233Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-16T09:00:04.368 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.349561814Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=694.523µs 2024-09-16T09:00:04.368 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.349986807Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-16T09:00:04.368 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.350678623Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=692.113µs 2024-09-16T09:00:04.502 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[29803]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:04.503 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.503 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.503 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.503 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.503 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[40056]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:04.503 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.503 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.503 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.503 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:04 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.351081121Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.351690142Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=609.17µs 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.352203018Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.352912039Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=708.638µs 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.353293428Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.35401578Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=722.238µs 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.354399465Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.354786816Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=385.85µs 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.355253927Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.355418571Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=165.107µs 2024-09-16T09:00:04.596 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.355829364Z level=info msg="Executing migration" id="create tag table" 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.356311765Z level=info msg="Migration successfully executed" id="create tag table" duration=482.241µs 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.358603377Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.359184736Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=579.999µs 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.361447197Z level=info msg="Executing migration" id="create login attempt table" 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.361942253Z level=info msg="Migration successfully executed" id="create login attempt table" duration=494.79µs 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.364064883Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.364603455Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=538.483µs 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.364955429Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.365493312Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=537.801µs 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.367786559Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.377423547Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=9.636764ms 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.37781916Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.378310981Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=491.897µs 2024-09-16T09:00:04.597 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.380752196Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.381336102Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=583.704µs 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.381740328Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.381959184Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=218.61µs 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.384578265Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.385052527Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=477.032µs 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.387080632Z level=info msg="Executing migration" id="create user auth table" 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.387584065Z level=info msg="Migration successfully executed" id="create user auth table" duration=502.146µs 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.390055697Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.39064726Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=591.379µs 2024-09-16T09:00:04.598 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.391010477Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.391048209Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=38.239µs 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.39375607Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.396672437Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=2.915928ms 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.397006028Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.399858195Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=2.851801ms 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.400217963Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.403118012Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=2.898751ms 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.403471185Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.406373126Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=2.901566ms 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.406732139Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-16T09:00:04.600 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.407279149Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=546.891µs 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.407624225Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.410529588Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=2.904902ms 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.410883825Z level=info msg="Executing migration" id="create server_lock table" 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.411381615Z level=info msg="Migration successfully executed" id="create server_lock table" duration=497.556µs 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.411747443Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.412296762Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=549.289µs 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.412651378Z level=info msg="Executing migration" id="create user auth token table" 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.41319264Z level=info msg="Migration successfully executed" id="create user auth token table" duration=540.288µs 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.413534339Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.414128091Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=593.586µs 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.414470805Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.415030967Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=559.966µs 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.415364454Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-16T09:00:04.601 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.415995088Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=630.405µs 2024-09-16T09:00:04.602 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.416335224Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-16T09:00:04.602 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.419400984Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.065448ms 2024-09-16T09:00:04.602 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.41974486Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-16T09:00:04.602 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.420311598Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=566.858µs 2024-09-16T09:00:04.602 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.420663413Z level=info msg="Executing migration" id="create cache_data table" 2024-09-16T09:00:04.602 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.421200323Z level=info msg="Migration successfully executed" id="create cache_data table" duration=537.334µs 2024-09-16T09:00:04.602 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.421535532Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-16T09:00:04.602 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.422107653Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=571.978µs 2024-09-16T09:00:04.602 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.422450555Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.423017579Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=566.98µs 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.423372831Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.423981243Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=608.033µs 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.424315497Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.42435227Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=37.289µs 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.424743703Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.424801913Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=57.098µs 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.425165248Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.425724382Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=558.784µs 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.42606079Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.426656771Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=594.723µs 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.427060817Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-16T09:00:04.603 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.427656752Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=595.469µs 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.427972999Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.428010527Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=38.577µs 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.428383654Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.428956531Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=572.719µs 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.429294729Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.429873975Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=579.159µs 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.43021232Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.430813089Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=600.338µs 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.431186407Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.431793306Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=606.45µs 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.433881904Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-16T09:00:04.604 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.437203301Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.319898ms 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.43756996Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.43822549Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=655.565µs 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.43857596Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.438642237Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=61.872µs 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.43902395Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.439585249Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=560.869µs 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.439935691Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.440563837Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=627.818µs 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.442618546Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.44327365Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=654.889µs 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.445405543Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.445460069Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=55.036µs 2024-09-16T09:00:04.605 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.44587187Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.446483633Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=611.417µs 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.446849053Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.447449255Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=599.722µs 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.449603897Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.450256999Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=652.354µs 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.452423711Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.453170313Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=747.5µs 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.454922762Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.458289463Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.366872ms 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.458641177Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.459226728Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=583.345µs 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.459573264Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.460186653Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=613.822µs 2024-09-16T09:00:04.606 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.46245398Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.479689082Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=17.23326ms 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.480066833Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.496158841Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=16.091741ms 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.496553733Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.49717812Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=623.056µs 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.499279846Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.499915123Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=635.136µs 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.502138541Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.505290274Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.151453ms 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.505687346Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.508884133Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=3.196415ms 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.509280927Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.509914979Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=634.657µs 2024-09-16T09:00:04.607 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.512045605Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.512713343Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=667.369µs 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.514800484Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.515558538Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=759.977µs 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.515946733Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.516667257Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=720.575µs 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.517045788Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.517087075Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=42.153µs 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.517859589Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.521268317Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=3.40892ms 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.521698118Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.525005701Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.307394ms 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.525369283Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.528693982Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.324143ms 2024-09-16T09:00:04.608 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.529064622Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.52965146Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=586.829µs 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.530014744Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.530653919Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=639.597µs 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.531013437Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.534320819Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.307674ms 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.534717233Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.538072293Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.355584ms 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.538435635Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.539073916Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=637.924µs 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.539452481Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.542749974Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.297109ms 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.543111301Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-16T09:00:04.609 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.546452286Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=3.341242ms 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.546860381Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.546898243Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=38.197µs 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.547702166Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.548399834Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=697.173µs 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.54876827Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.549395371Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=626.713µs 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.549740154Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.550419405Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=679.476µs 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.550804482Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.550842544Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=39.456µs 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.55183139Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.555417004Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=3.589069ms 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.555776005Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-16T09:00:04.610 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.559260817Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=3.485126ms 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.559614134Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.563116956Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=3.502565ms 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.563496588Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.567008111Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=3.511256ms 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.567381906Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.570732282Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.350158ms 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.571084128Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.571121877Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=39.21µs 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.571950185Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.572444727Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=488.381µs 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.572781809Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.576379227Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.597051ms 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.576730338Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-16T09:00:04.611 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.576768104Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=38.271µs 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.577617946Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.581189513Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=3.571239ms 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.58156343Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.582167772Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=604.1µs 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.582547832Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.586143048Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=3.594012ms 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.586511995Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.587020844Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=509.802µs 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.587418497Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.588029455Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=610.977µs 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.588412851Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.592048042Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.630319ms 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.592435589Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.592943661Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=507.998µs 2024-09-16T09:00:04.612 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.595067351Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-16T09:00:04.613 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.595850967Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=782.81µs 2024-09-16T09:00:04.613 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.597769389Z level=info msg="Executing migration" id="create alert_image table" 2024-09-16T09:00:04.613 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.598312418Z level=info msg="Migration successfully executed" id="create alert_image table" duration=542.771µs 2024-09-16T09:00:04.846 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.600696327Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-16T09:00:04.846 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.601407885Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=711.352µs 2024-09-16T09:00:04.846 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.603439918Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-16T09:00:04.846 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.603486609Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=47.081µs 2024-09-16T09:00:04.846 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.606253412Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-16T09:00:04.846 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.606886848Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=633.557µs 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.608861538Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.609441085Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=579.535µs 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.611814753Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.61203785Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.614406261Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.614676332Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=270.025µs 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.617471692Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.618193679Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=723.542µs 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.620391567Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.624186082Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=3.796554ms 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.624553493Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-16T09:00:04.847 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.625242854Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=689.136µs 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.627598421Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.628368074Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=771.543µs 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.630129828Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.630700775Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=570.661µs 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.632864544Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.63351327Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=648.468µs 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.635698696Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.636311597Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=611.143µs 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.638444646Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.638477155Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=33.178µs 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.641156568Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.641211049Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=54.732µs 2024-09-16T09:00:04.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.643890246Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-16T09:00:04.849 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.644118974Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=231.037µs 2024-09-16T09:00:04.849 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.646522741Z level=info msg="Executing migration" id="create data_keys table" 2024-09-16T09:00:04.849 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.647197628Z level=info msg="Migration successfully executed" id="create data_keys table" duration=674.499µs 2024-09-16T09:00:04.850 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.649292638Z level=info msg="Executing migration" id="create secrets table" 2024-09-16T09:00:04.850 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.649865106Z level=info msg="Migration successfully executed" id="create secrets table" duration=573.312µs 2024-09-16T09:00:04.850 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.65205835Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-16T09:00:04.850 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.672944348Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=20.886837ms 2024-09-16T09:00:04.850 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.673334887Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-16T09:00:04.850 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.677267414Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=3.932526ms 2024-09-16T09:00:04.850 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.677664117Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-16T09:00:04.850 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.677780901Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=115.815µs 2024-09-16T09:00:04.850 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.680425649Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.701099193Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=20.673067ms 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.701490505Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.722415113Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=20.924335ms 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.722820909Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.723398899Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=577.889µs 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.725664674Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.726335286Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=670.526µs 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.728342072Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.728510149Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=168.114µs 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.730937852Z level=info msg="Executing migration" id="create permission table" 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.731564447Z level=info msg="Migration successfully executed" id="create permission table" duration=626.393µs 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.733851768Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-16T09:00:04.851 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.734519982Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=670.73µs 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.736455125Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.737153729Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=698.378µs 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.739382613Z level=info msg="Executing migration" id="create role table" 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.740031677Z level=info msg="Migration successfully executed" id="create role table" duration=651.369µs 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.741916942Z level=info msg="Executing migration" id="add column display_name" 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.746151738Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.235463ms 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.746537971Z level=info msg="Executing migration" id="add column group_name" 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.750579444Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.041364ms 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.750975544Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.751638441Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=654.126µs 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.752024945Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.752716447Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=691.199µs 2024-09-16T09:00:04.852 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.753084036Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.753765756Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=681.436µs 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.75414376Z level=info msg="Executing migration" id="create team role table" 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.754711609Z level=info msg="Migration successfully executed" id="create team role table" duration=567.424µs 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.755082361Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.75577054Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=688.017µs 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.756185315Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.756916048Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=716.026µs 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.757304279Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.757973976Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=669.58µs 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.758356697Z level=info msg="Executing migration" id="create user role table" 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.7589378Z level=info msg="Migration successfully executed" id="create user role table" duration=580.95µs 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.759321701Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-16T09:00:04.853 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.759997327Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=674.767µs 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.760367806Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.761040961Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=672.948µs 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.761414022Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.762078199Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=661.915µs 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.762444143Z level=info msg="Executing migration" id="create builtin role table" 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.763006251Z level=info msg="Migration successfully executed" id="create builtin role table" duration=562.116µs 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.763383485Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.764064432Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=680.836µs 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.764431851Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.765154498Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=722.371µs 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.765524896Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.770041571Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=4.516477ms 2024-09-16T09:00:04.854 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.770412636Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.77108389Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=671.203µs 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.771461741Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.772145346Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=683.347µs 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.772512382Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.773179368Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=665.929µs 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.773537898Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.77421369Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=675.591µs 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.774575457Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.775092187Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=517.216µs 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.775465367Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.776133564Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=667.911µs 2024-09-16T09:00:04.855 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.776488742Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.780992338Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=4.503333ms 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.781357088Z level=info msg="Executing migration" id="permission kind migration" 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.785933356Z level=info msg="Migration successfully executed" id="permission kind migration" duration=4.575926ms 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.786322462Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.790727981Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.405186ms 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.791132787Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.795524908Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=4.391687ms 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.795902163Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.796590741Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=688.458µs 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.796996592Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.797762315Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=764.385µs 2024-09-16T09:00:04.856 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.798124997Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.798789511Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=664.555µs 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.799196667Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.799797057Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=600.191µs 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.800150054Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.800827089Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=676.781µs 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.801188758Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.801234022Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=44.996µs 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.802206763Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.80222598Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=19.976µs 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.803020123Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.803444893Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=427.323µs 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.803849117Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-16T09:00:04.857 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.804183781Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=335.26µs 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.804700874Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.805112266Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=411.437µs 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.807677233Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.807809143Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=132.034µs 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.810196879Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.810498045Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=300.976µs 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.813273454Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.813792062Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=518.367µs 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.816031474Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.816828857Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=799.223µs 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.81872623Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.823295092Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=4.568533ms 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.823640724Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-16T09:00:04.858 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.823678996Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=38.806µs 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.826442707Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.827144982Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=703.87µs 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.829079991Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.829738679Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=658.614µs 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.831805363Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.832444628Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=639.054µs 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.834604687Z level=info msg="Executing migration" id="add correlation config column" 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.839402076Z level=info msg="Migration successfully executed" id="add correlation config column" duration=4.79717ms 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.839745185Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.840383021Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=637.879µs 2024-09-16T09:00:04.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.842607392Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-16T09:00:04.860 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.843271475Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=664.086µs 2024-09-16T09:00:04.860 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.845434742Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-16T09:00:04.869 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:04 smithi111 systemd[1]: Starting Ceph node-exporter.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:00:05.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.858934438Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=13.500033ms 2024-09-16T09:00:05.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.85929281Z level=info msg="Executing migration" id="create correlation v2" 2024-09-16T09:00:05.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.860024536Z level=info msg="Migration successfully executed" id="create correlation v2" duration=731.368µs 2024-09-16T09:00:05.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.862214019Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-16T09:00:05.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.86301213Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=800.757µs 2024-09-16T09:00:05.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.864808177Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-16T09:00:05.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.865498147Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=689.769µs 2024-09-16T09:00:05.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.867590044Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-16T09:00:05.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.868230992Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=640.649µs 2024-09-16T09:00:05.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.870382109Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.870559019Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=176.472µs 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.87293352Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.873463077Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=529.338µs 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.875717126Z level=info msg="Executing migration" id="add provisioning column" 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.880410515Z level=info msg="Migration successfully executed" id="add provisioning column" duration=4.69314ms 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.880767604Z level=info msg="Executing migration" id="create entity_events table" 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.88127791Z level=info msg="Migration successfully executed" id="create entity_events table" duration=508.649µs 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.883491431Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.884170977Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=679.563µs 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.886204223Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.886483181Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-16T09:00:05.098 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.889049864Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.889354716Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.891852929Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.89237283Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=519.713µs 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.894381892Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.895048144Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=666.076µs 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.897326497Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.898006134Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=679.429µs 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.900099931Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.900792993Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=692.555µs 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.902759938Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.903404914Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=644.898µs 2024-09-16T09:00:05.099 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.905442203Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-16T09:00:05.100 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.906115077Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=672.792µs 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.908084734Z level=info msg="Executing migration" id="Drop public config table" 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.908614882Z level=info msg="Migration successfully executed" id="Drop public config table" duration=531.148µs 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.910949016Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.911625104Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=675.924µs 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.91361478Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.914280637Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=665.712µs 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.916381571Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.917099091Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=717.343µs 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.917443762Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.918118555Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=675.608µs 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.918469376Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.934487578Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=16.017859ms 2024-09-16T09:00:05.101 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.934860499Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.93988523Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.024384ms 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.940254466Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.94504578Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=4.791011ms 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.945399035Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.945534685Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=135.638µs 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.946217238Z level=info msg="Executing migration" id="add share column" 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.951021831Z level=info msg="Migration successfully executed" id="add share column" duration=4.804334ms 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.951376634Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.951504974Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=128.398µs 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.952396036Z level=info msg="Executing migration" id="create file table" 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.953056895Z level=info msg="Migration successfully executed" id="create file table" duration=665.501µs 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.953413679Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.954079163Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=665.358µs 2024-09-16T09:00:05.102 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.954433352Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-16T09:00:05.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.955096592Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=662.972µs 2024-09-16T09:00:05.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.95545273Z level=info msg="Executing migration" id="create file_meta table" 2024-09-16T09:00:05.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.955958988Z level=info msg="Migration successfully executed" id="create file_meta table" duration=506.006µs 2024-09-16T09:00:05.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.956311919Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-16T09:00:05.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.956990931Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=678.653µs 2024-09-16T09:00:05.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.957367492Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-16T09:00:05.103 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.95740579Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=39.006µs 2024-09-16T09:00:05.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.95834086Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-16T09:00:05.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.958438508Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=98.665µs 2024-09-16T09:00:05.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.959046443Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-16T09:00:05.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.959376981Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=330.321µs 2024-09-16T09:00:05.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.961851579Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-16T09:00:05.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.961978335Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=126.797µs 2024-09-16T09:00:05.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.96470786Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-16T09:00:05.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.965491922Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=783.811µs 2024-09-16T09:00:05.104 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.967698435Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.97255239Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=4.855932ms 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.972921245Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.973023514Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=103.93µs 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.975766364Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.976499991Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=733.196µs 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.978709751Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.978971106Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=261.913µs 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.981331832Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.9814726Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=140.754µs 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.984208996Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.984502216Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=293.146µs 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.984891629Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-16T09:00:05.105 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.989716466Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=4.824486ms 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.990053603Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.994876675Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=4.822585ms 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.995244639Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.995909317Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=664.589µs 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:04 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:04.998253598Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.053503499Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=55.251512ms 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.053890552Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.054694462Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=802.286µs 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.056873613Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.057562508Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=688.704µs 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.059534976Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.07615545Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=16.622532ms 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.07653489Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-16T09:00:05.106 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.081393487Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=4.858425ms 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.081758906Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.081943012Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=182.857µs 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.084452722Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.084572646Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=119.933µs 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.087372137Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.087676232Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=307.863µs 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.089837156Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.089957137Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=119.97µs 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.090331705Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.090451757Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=120.074µs 2024-09-16T09:00:05.107 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.093162924Z level=info msg="Executing migration" id="create folder table" 2024-09-16T09:00:05.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.093856576Z level=info msg="Migration successfully executed" id="create folder table" duration=694.755µs 2024-09-16T09:00:05.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.095906409Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-16T09:00:05.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.096807698Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=903.058µs 2024-09-16T09:00:05.108 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.098438693Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-16T09:00:05.175 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:04 smithi111 bash[74829]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-16T09:00:05.349 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-mon[33315]: Deploying daemon node-exporter.a on smithi111 2024-09-16T09:00:05.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.09915348Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=714.651µs 2024-09-16T09:00:05.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.099495895Z level=info msg="Executing migration" id="Update folder title length" 2024-09-16T09:00:05.349 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.099511212Z level=info msg="Migration successfully executed" id="Update folder title length" duration=15.968µs 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.102229772Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.102948927Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=718.73µs 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.104913657Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.105572835Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=659.35µs 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.107687713Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.108410401Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=722.242µs 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.108774966Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.10906342Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=288.377µs 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.111504048Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.111705512Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=202.348µs 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.114150634Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-16T09:00:05.350 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.114802368Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=651.462µs 2024-09-16T09:00:05.351 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.116843304Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-16T09:00:05.351 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.117541692Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=698.152µs 2024-09-16T09:00:05.351 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.117895611Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-16T09:00:05.351 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.118553241Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=657.576µs 2024-09-16T09:00:05.351 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.12075826Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-16T09:00:05.351 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.121484326Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=725.805µs 2024-09-16T09:00:05.351 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.123428967Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-16T09:00:05.351 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.124108747Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=680.965µs 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.124461295Z level=info msg="Executing migration" id="create anon_device table" 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.125026945Z level=info msg="Migration successfully executed" id="create anon_device table" duration=565.311µs 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.125398004Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.126159732Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=761.373µs 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.126527869Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.127243404Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=715.382µs 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.127624225Z level=info msg="Executing migration" id="create signing_key table" 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.128241982Z level=info msg="Migration successfully executed" id="create signing_key table" duration=617.591µs 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.128604253Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.129293319Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=688.972µs 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.129651076Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.130357477Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=706.332µs 2024-09-16T09:00:05.352 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.130692334Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.130854768Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=164.385µs 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.131511712Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.137170528Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.658558ms 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.137536331Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.138015274Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=479.616µs 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.13842092Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.139115366Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=694.201µs 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.139473769Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.140142265Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=669.402µs 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.140474196Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.141229305Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=754.946µs 2024-09-16T09:00:05.353 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.141586342Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-16T09:00:05.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.142364486Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=777.808µs 2024-09-16T09:00:05.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.142706118Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-16T09:00:05.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.143402073Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=695.736µs 2024-09-16T09:00:05.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.143753136Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-16T09:00:05.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.144393785Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=640.611µs 2024-09-16T09:00:05.354 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.144759042Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.145211153Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=452.744µs 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.145592733Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.145760233Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=168.075µs 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.146432912Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.146470788Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=39.899µs 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.146875717Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.151980489Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.104468ms 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.152334191Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.157571031Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.236467ms 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.157926737Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.158130515Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=204.154µs 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=migrator t=2024-09-16T09:00:05.158476676Z level=info msg="migrations completed" performed=547 skipped=0 duration=1.27888724s 2024-09-16T09:00:05.355 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore t=2024-09-16T09:00:05.159300429Z level=info msg="Created default organization" 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=secrets t=2024-09-16T09:00:05.159806132Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=plugin.store t=2024-09-16T09:00:05.173910909Z level=info msg="Loading plugins..." 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=local.finder t=2024-09-16T09:00:05.21325574Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=plugin.store t=2024-09-16T09:00:05.213272082Z level=info msg="Plugins loaded" count=55 duration=39.361601ms 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=query_data t=2024-09-16T09:00:05.21547898Z level=info msg="Query Service initialization" 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=live.push_http t=2024-09-16T09:00:05.220727296Z level=info msg="Live Push Gateway initialization" 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.migration t=2024-09-16T09:00:05.22208438Z level=info msg=Starting 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.migration t=2024-09-16T09:00:05.22237524Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.migration orgID=1 t=2024-09-16T09:00:05.222672268Z level=info msg="Migrating alerts for organisation" 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.migration orgID=1 t=2024-09-16T09:00:05.22312369Z level=info msg="Alerts found to migrate" alerts=0 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.migration t=2024-09-16T09:00:05.22436699Z level=info msg="Completed alerting migration" 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.state.manager t=2024-09-16T09:00:05.235180482Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-16T09:00:05.356 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=infra.usagestats.collector t=2024-09-16T09:00:05.236511368Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=provisioning.datasources t=2024-09-16T09:00:05.238146553Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=provisioning.datasources t=2024-09-16T09:00:05.250282483Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=provisioning.alerting t=2024-09-16T09:00:05.262699313Z level=info msg="starting to provision alerting" 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=provisioning.alerting t=2024-09-16T09:00:05.262710274Z level=info msg="finished to provision alerting" 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=grafanaStorageLogger t=2024-09-16T09:00:05.262816186Z level=info msg="Storage starting" 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.state.manager t=2024-09-16T09:00:05.262861964Z level=info msg="Warming state cache for startup" 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.state.manager t=2024-09-16T09:00:05.263084147Z level=info msg="State cache has been initialized" states=0 duration=221.503µs 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.multiorg.alertmanager t=2024-09-16T09:00:05.263150783Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ngalert.scheduler t=2024-09-16T09:00:05.263199852Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ticker t=2024-09-16T09:00:05.263242237Z level=info msg=starting first_tick=2024-09-16T09:00:10Z 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=http.server t=2024-09-16T09:00:05.265931864Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=http.server t=2024-09-16T09:00:05.266195929Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=provisioning.dashboard t=2024-09-16T09:00:05.297605894Z level=info msg="starting to provision dashboards" 2024-09-16T09:00:05.357 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore.transactions t=2024-09-16T09:00:05.327540794Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-16T09:00:05.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=plugins.update.checker t=2024-09-16T09:00:05.332065614Z level=info msg="Update check succeeded" duration=69.056215ms 2024-09-16T09:00:05.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore.transactions t=2024-09-16T09:00:05.338356374Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-16T09:00:05.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=grafana.update.checker t=2024-09-16T09:00:05.341701393Z level=info msg="Update check succeeded" duration=78.764963ms 2024-09-16T09:00:05.358 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore.transactions t=2024-09-16T09:00:05.348768954Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-16T09:00:05.664 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore.transactions t=2024-09-16T09:00:05.359519284Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-09-16T09:00:05.664 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore.transactions t=2024-09-16T09:00:05.369901063Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-09-16T09:00:05.664 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=provisioning.dashboard t=2024-09-16T09:00:05.433572231Z level=info msg="finished to provision dashboards" 2024-09-16T09:00:05.664 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=grafana-apiserver t=2024-09-16T09:00:05.449568116Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-16T09:00:05.665 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:00:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=grafana-apiserver t=2024-09-16T09:00:05.449948393Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-16T09:00:05.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:05 smithi111 ceph-mon[29803]: Deploying daemon node-exporter.a on smithi111 2024-09-16T09:00:05.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:05 smithi111 ceph-mon[40056]: Deploying daemon node-exporter.a on smithi111 2024-09-16T09:00:05.676 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:05 smithi111 bash[74829]: Getting image source signatures 2024-09-16T09:00:05.676 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:05 smithi111 bash[74829]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-16T09:00:05.676 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:05 smithi111 bash[74829]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-16T09:00:05.676 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:05 smithi111 bash[74829]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-16T09:00:06.569 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:06 smithi111 ceph-mon[29803]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:06.569 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:06 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:06.570 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:06 smithi111 ceph-mon[40056]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:06.570 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:06 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:06.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:06 smithi114 ceph-mon[33315]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:06 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:06.926 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:06 smithi111 bash[74829]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-16T09:00:06.926 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:06 smithi111 bash[74829]: Writing manifest to image destination 2024-09-16T09:00:07.353 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 podman[74829]: 2024-09-16 09:00:07.018156566 +0000 UTC m=+2.240996969 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-16T09:00:07.353 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 podman[74829]: 2024-09-16 09:00:07.12536827 +0000 UTC m=+2.348208742 container create 0ffad191150764cf1a630a34475ce5e0a2122c4a257c58d53d5318a715167f49 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:07.677 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 podman[74829]: 2024-09-16 09:00:07.353229306 +0000 UTC m=+2.576069710 container init 0ffad191150764cf1a630a34475ce5e0a2122c4a257c58d53d5318a715167f49 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:07.677 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 podman[74829]: 2024-09-16 09:00:07.35817087 +0000 UTC m=+2.581011283 container start 0ffad191150764cf1a630a34475ce5e0a2122c4a257c58d53d5318a715167f49 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:07.677 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.361Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-16T09:00:07.678 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.361Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-16T09:00:07.678 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.361Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-16T09:00:07.678 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-16T09:00:07.678 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-16T09:00:07.678 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-16T09:00:07.679 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-16T09:00:07.679 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=arp 2024-09-16T09:00:07.679 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-16T09:00:07.679 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-16T09:00:07.679 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-16T09:00:07.679 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-16T09:00:07.680 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-16T09:00:07.680 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-16T09:00:07.680 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-16T09:00:07.680 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-16T09:00:07.680 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=edac 2024-09-16T09:00:07.681 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-16T09:00:07.681 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-16T09:00:07.681 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-16T09:00:07.681 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-16T09:00:07.681 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-16T09:00:07.681 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-16T09:00:07.682 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-16T09:00:07.682 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-16T09:00:07.682 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-16T09:00:07.682 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-16T09:00:07.682 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-16T09:00:07.683 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-16T09:00:07.683 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-16T09:00:07.683 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-16T09:00:07.683 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-16T09:00:07.683 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-16T09:00:07.683 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=os 2024-09-16T09:00:07.684 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-16T09:00:07.684 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-16T09:00:07.684 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-16T09:00:07.684 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-16T09:00:07.684 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-16T09:00:07.684 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-16T09:00:07.685 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-16T09:00:07.685 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=stat 2024-09-16T09:00:07.685 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-16T09:00:07.685 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-16T09:00:07.685 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-16T09:00:07.685 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=time 2024-09-16T09:00:07.686 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-16T09:00:07.686 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=uname 2024-09-16T09:00:07.686 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-16T09:00:07.686 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-16T09:00:07.686 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.362Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-16T09:00:07.686 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.363Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-16T09:00:07.687 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a[74905]: ts=2024-09-16T09:00:07.363Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-16T09:00:07.687 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 bash[74829]: 0ffad191150764cf1a630a34475ce5e0a2122c4a257c58d53d5318a715167f49 2024-09-16T09:00:07.687 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:00:07 smithi111 systemd[1]: Started Ceph node-exporter.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:00:08.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:08 smithi114 ceph-mon[33315]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:08 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:08 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:08 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:08.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:08 smithi114 bash[63377]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-16T09:00:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:08 smithi111 ceph-mon[29803]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:08 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:08 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:08 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:08.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:08 smithi111 ceph-mon[40056]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:08.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:08 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:08 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:08 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:09.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:09 smithi114 ceph-mon[33315]: Deploying daemon node-exporter.b on smithi114 2024-09-16T09:00:09.663 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:09 smithi114 bash[63377]: Getting image source signatures 2024-09-16T09:00:09.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:09 smithi114 bash[63377]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-16T09:00:09.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:09 smithi114 bash[63377]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-16T09:00:09.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:09 smithi114 bash[63377]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-16T09:00:09.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:09 smithi111 ceph-mon[29803]: Deploying daemon node-exporter.b on smithi114 2024-09-16T09:00:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:09 smithi111 ceph-mon[40056]: Deploying daemon node-exporter.b on smithi114 2024-09-16T09:00:10.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:10 smithi114 ceph-mon[33315]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:10.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:10 smithi111 ceph-mon[29803]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:10 smithi111 ceph-mon[40056]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:10.913 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:10 smithi114 bash[63377]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-16T09:00:10.914 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:10 smithi114 bash[63377]: Writing manifest to image destination 2024-09-16T09:00:11.373 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 podman[63377]: 2024-09-16 09:00:11.02093659 +0000 UTC m=+2.689532906 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-16T09:00:11.373 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 podman[63377]: 2024-09-16 09:00:11.094579409 +0000 UTC m=+2.763175768 container create b37ec837e20effbf4d0fcde451371946b0a6b9d77d7c9fe6ff077587e387d8af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T09:00:11.373 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 podman[63377]: 2024-09-16 09:00:11.372294599 +0000 UTC m=+3.040890939 container init b37ec837e20effbf4d0fcde451371946b0a6b9d77d7c9fe6ff077587e387d8af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T09:00:11.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 podman[63377]: 2024-09-16 09:00:11.378198283 +0000 UTC m=+3.046794601 container start b37ec837e20effbf4d0fcde451371946b0a6b9d77d7c9fe6ff077587e387d8af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.381Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.381Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.382Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.382Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.382Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.382Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=arp 2024-09-16T09:00:11.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-16T09:00:11.665 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-16T09:00:11.665 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-16T09:00:11.665 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-16T09:00:11.665 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-16T09:00:11.665 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-16T09:00:11.665 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-16T09:00:11.665 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-16T09:00:11.665 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=edac 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-16T09:00:11.666 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=os 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-16T09:00:11.667 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=stat 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=time 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=uname 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b[63450]: ts=2024-09-16T09:00:11.383Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-16T09:00:11.668 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 bash[63377]: b37ec837e20effbf4d0fcde451371946b0a6b9d77d7c9fe6ff077587e387d8af 2024-09-16T09:00:11.669 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:00:11 smithi114 systemd[1]: Started Ceph node-exporter.b for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:00:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:12 smithi114 ceph-mon[33315]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:12 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:12 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:12 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:12 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:12 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:12 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:00:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[29803]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:00:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[40056]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:12 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:00:13.371 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:13.463 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:13.463 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:00:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:00:13.463 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:14.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:14 smithi111 ceph-mon[29803]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:14.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:14 smithi111 ceph-mon[40056]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:14.569 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:14 smithi114 ceph-mon[33315]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:15 smithi114 ceph-mon[33315]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:15 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:15 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:15.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:15 smithi111 ceph-mon[29803]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:15.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:15 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:15.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:15 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:15 smithi111 ceph-mon[40056]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:15 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:15 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.075 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.075 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[29803]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[29803]: Reconfiguring daemon alertmanager.a on smithi111 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:00:17.076 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:00:17.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:17.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[40056]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-16T09:00:17.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:16 smithi111 ceph-mon[40056]: Reconfiguring daemon alertmanager.a on smithi111 2024-09-16T09:00:17.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:00:17.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:00:17.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:17.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:17.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:16 smithi114 ceph-mon[33315]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-16T09:00:17.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:16 smithi114 ceph-mon[33315]: Reconfiguring daemon alertmanager.a on smithi111 2024-09-16T09:00:18.125 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:17 smithi111 ceph-mon[29803]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:18.125 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:17 smithi111 ceph-mon[40056]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:18.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:17 smithi114 ceph-mon[33315]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:19.426 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:19 smithi111 systemd[1]: Stopping Ceph alertmanager.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:00:19.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:19 smithi111 ceph-mon[29803]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:19.903 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:19 smithi111 ceph-mon[40056]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:19.903 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:19 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[74572]: ts=2024-09-16T09:00:19.562Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-16T09:00:19.903 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:19 smithi111 podman[75612]: 2024-09-16 09:00:19.583625416 +0000 UTC m=+0.156201437 container died 13e7816bc96cd7c4fcdb3401ff2eee1e75ac1b2b651b00e3f5f6127ce4b543d6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:20.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:19 smithi114 ceph-mon[33315]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:20.175 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:20 smithi111 podman[75612]: 2024-09-16 09:00:20.117803347 +0000 UTC m=+0.690379360 container cleanup 13e7816bc96cd7c4fcdb3401ff2eee1e75ac1b2b651b00e3f5f6127ce4b543d6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:20.176 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:20 smithi111 bash[75612]: ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a 2024-09-16T09:00:20.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:20 smithi111 podman[75629]: 2024-09-16 09:00:20.327674922 +0000 UTC m=+0.738543762 container remove 13e7816bc96cd7c4fcdb3401ff2eee1e75ac1b2b651b00e3f5f6127ce4b543d6 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:20.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:20 smithi111 podman[75629]: 2024-09-16 09:00:20.411704274 +0000 UTC m=+0.822573112 volume remove 0804d2f8221f110dedb704b2e6bc28d0f80916a8dbb2d4f4a445092a4b996b7b 2024-09-16T09:00:21.176 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:20 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-16T09:00:21.176 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:20 smithi111 systemd[1]: Stopped Ceph alertmanager.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:00:21.176 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:20 smithi111 systemd[1]: Starting Ceph alertmanager.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:00:21.176 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:20 smithi111 podman[75724]: 2024-09-16 09:00:20.884929012 +0000 UTC m=+0.017378673 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-16T09:00:21.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:00:21 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:00:22.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:21 smithi114 ceph-mon[33315]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:22.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:21 smithi111 ceph-mon[29803]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:22.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:21 smithi111 ceph-mon[40056]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:22.176 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:21 smithi111 podman[75724]: 2024-09-16 09:00:21.88833055 +0000 UTC m=+1.020780206 volume create 4d947a41014c4b82e6ddc9e11c220f40e4b8c0755e98bc67a2b7b3542bcde64f 2024-09-16T09:00:23.062 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:23.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:23.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:23.176 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:22 smithi111 podman[75724]: 2024-09-16 09:00:22.789556623 +0000 UTC m=+1.922006275 container create 553e60a063af2e2ccd5d7220ba0fc3073b2d8572974094b4c4eefd5e4e61334b (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:23.675 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:00:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:00:24.107 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:23 smithi111 ceph-mon[29803]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:24.108 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:23 smithi111 ceph-mon[40056]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:24.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:23 smithi114 ceph-mon[33315]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:24.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 podman[75724]: 2024-09-16 09:00:24.22759741 +0000 UTC m=+3.360047065 container init 553e60a063af2e2ccd5d7220ba0fc3073b2d8572974094b4c4eefd5e4e61334b (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:24.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 podman[75724]: 2024-09-16 09:00:24.232461145 +0000 UTC m=+3.364910804 container start 553e60a063af2e2ccd5d7220ba0fc3073b2d8572974094b4c4eefd5e4e61334b (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:24.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:24.246Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-16T09:00:24.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:24.246Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-16T09:00:24.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:24.247Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.111 port=9094 2024-09-16T09:00:24.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:24.248Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-16T09:00:24.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:24.279Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-16T09:00:24.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:24.280Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-16T09:00:24.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:24.281Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-16T09:00:24.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:24.281Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-16T09:00:24.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 bash[75724]: 553e60a063af2e2ccd5d7220ba0fc3073b2d8572974094b4c4eefd5e4e61334b 2024-09-16T09:00:24.677 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:24 smithi111 systemd[1]: Started Ceph alertmanager.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:00:25.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:25 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:25 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:25 smithi114 ceph-mon[33315]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-16T09:00:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:25 smithi114 ceph-mon[33315]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:25 smithi114 ceph-mon[33315]: Reconfiguring daemon prometheus.a on smithi114 2024-09-16T09:00:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[29803]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-16T09:00:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[29803]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[29803]: Reconfiguring daemon prometheus.a on smithi114 2024-09-16T09:00:25.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:25.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:25.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[40056]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-16T09:00:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[40056]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:25 smithi111 ceph-mon[40056]: Reconfiguring daemon prometheus.a on smithi114 2024-09-16T09:00:26.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:26 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:26.249Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000939183s 2024-09-16T09:00:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-mon[33315]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:27.664 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 systemd[1]: Stopping Ceph prometheus.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:00:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:27 smithi111 ceph-mon[29803]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:27 smithi111 ceph-mon[40056]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:28.009 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.716Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-16T09:00:28.009 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.717Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-16T09:00:28.009 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.717Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-16T09:00:28.009 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.717Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-16T09:00:28.009 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.717Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-16T09:00:28.009 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.717Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-16T09:00:28.009 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.717Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-16T09:00:28.010 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.717Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-16T09:00:28.010 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.717Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-16T09:00:28.010 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.718Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-16T09:00:28.010 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.718Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-16T09:00:28.010 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[61540]: ts=2024-09-16T09:00:27.718Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-16T09:00:28.010 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:27 smithi114 podman[64284]: 2024-09-16 09:00:27.727327185 +0000 UTC m=+0.111342909 container died 531f3e0837669b097cbee4d3b6198dda6398510c83d910fe152ab3a699c80b2a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:28.307 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:28 smithi114 podman[64284]: 2024-09-16 09:00:28.09768046 +0000 UTC m=+0.481696182 container cleanup 531f3e0837669b097cbee4d3b6198dda6398510c83d910fe152ab3a699c80b2a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:28.307 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:28 smithi114 bash[64284]: ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a 2024-09-16T09:00:28.663 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:28 smithi114 podman[64301]: 2024-09-16 09:00:28.307123606 +0000 UTC m=+0.581539212 container remove 531f3e0837669b097cbee4d3b6198dda6398510c83d910fe152ab3a699c80b2a (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:28.664 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:28 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-16T09:00:28.664 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:28 smithi114 systemd[1]: Stopped Ceph prometheus.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:00:28.664 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:28 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.067s CPU time. 2024-09-16T09:00:28.664 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:28 smithi114 systemd[1]: Starting Ceph prometheus.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:00:29.163 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:28 smithi114 podman[64403]: 2024-09-16 09:00:28.619561458 +0000 UTC m=+0.016514384 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-16T09:00:29.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:29 smithi114 ceph-mon[33315]: pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:29.664 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:29 smithi114 podman[64403]: 2024-09-16 09:00:29.375750886 +0000 UTC m=+0.772703807 container create 11c080ad8755ea4f8b5700e4bbb8cafbd990fa3fd885bf3623ddb397669163ce (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:29 smithi111 ceph-mon[29803]: pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:29.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:29 smithi111 ceph-mon[40056]: pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:30.827 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:30 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-16T09:00:30.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:30 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-16T09:00:30.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:30 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T09:00:30.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:30 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-16T09:00:30.829 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:30 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-16T09:00:30.829 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:30 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T09:00:30.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:30 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-16T09:00:30.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:30 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-16T09:00:30.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:30 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]} : dispatch 2024-09-16T09:00:31.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:00:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-mon[33315]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-16T09:00:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-16T09:00:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-16T09:00:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-mon[33315]: osdmap e64: 8 total, 8 up, 8 in 2024-09-16T09:00:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:31.915 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 podman[64403]: 2024-09-16 09:00:31.424399181 +0000 UTC m=+2.821352109 container init 11c080ad8755ea4f8b5700e4bbb8cafbd990fa3fd885bf3623ddb397669163ce (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:31.915 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 podman[64403]: 2024-09-16 09:00:31.430404214 +0000 UTC m=+2.827357137 container start 11c080ad8755ea4f8b5700e4bbb8cafbd990fa3fd885bf3623ddb397669163ce (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T09:00:31.915 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.452Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-16T09:00:31.915 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.452Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-16T09:00:31.915 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.452Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi114 (none))" 2024-09-16T09:00:31.915 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.452Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-16T09:00:31.915 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.452Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.454Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.454Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.456Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.456Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.458Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.458Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.808µs 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.458Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.458Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.458Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.458Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.426µs wal_replay_duration=480.606µs wbl_replay_duration=124ns total_replay_duration=525.193µs 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.460Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.460Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.460Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-16T09:00:31.916 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.476Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.366457ms db_storage=827ns remote_storage=1.549µs web_handler=355ns query_engine=571ns scrape=418.105µs scrape_sd=354.357µs notify=12.748µs notify_sd=8.504µs rules=15.199114ms tracing=6.269µs 2024-09-16T09:00:31.917 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.476Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-16T09:00:31.917 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:00:31.476Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-16T09:00:31.925 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[40056]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[40056]: osdmap e64: 8 total, 8 up, 8 in 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[29803]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[29803]: osdmap e64: 8 total, 8 up, 8 in 2024-09-16T09:00:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:32.836 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:32.836 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: osdmap e65: 8 total, 8 up, 8 in 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"} : dispatch 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"} : dispatch 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"} : dispatch 2024-09-16T09:00:32.837 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: osdmap e65: 8 total, 8 up, 8 in 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"} : dispatch 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"} : dispatch 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.838 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-16T09:00:32.839 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"} : dispatch 2024-09-16T09:00:32.839 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.839 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:32] ENGINE Bus STOPPING 2024-09-16T09:00:32.839 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T09:00:32.839 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:32] ENGINE Bus STOPPED 2024-09-16T09:00:32.839 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:32] ENGINE Bus STARTING 2024-09-16T09:00:32.839 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:32] ENGINE Serving on http://:::9283 2024-09-16T09:00:32.839 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:32] ENGINE Bus STARTED 2024-09-16T09:00:32.913 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:32 smithi114 bash[64403]: 11c080ad8755ea4f8b5700e4bbb8cafbd990fa3fd885bf3623ddb397669163ce 2024-09-16T09:00:32.914 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:00:32 smithi114 systemd[1]: Started Ceph prometheus.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: osdmap e65: 8 total, 8 up, 8 in 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"} : dispatch 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"} : dispatch 2024-09-16T09:00:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:32.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-16T09:00:32.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"} : dispatch 2024-09-16T09:00:32.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:33.176 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:32 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:32] ENGINE Bus STOPPING 2024-09-16T09:00:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-16T09:00:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"}]: dispatch 2024-09-16T09:00:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-16T09:00:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-09-16T09:00:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-16T09:00:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-09-16T09:00:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[29803]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:33] ENGINE Bus STOPPED 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:33] ENGINE Bus STARTING 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:33] ENGINE Serving on http://:::9283 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:33] ENGINE Bus STARTED 2024-09-16T09:00:33.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:33] ENGINE Bus STOPPING 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"}]: dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T09:00:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-mon[40056]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"}]: dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T09:00:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:33 smithi114 ceph-mon[33315]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-16T09:00:34.252 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T09:00:34.253 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:33] ENGINE Bus STOPPED 2024-09-16T09:00:34.253 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:33] ENGINE Bus STARTING 2024-09-16T09:00:34.253 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:34 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:34] ENGINE Serving on http://:::9283 2024-09-16T09:00:34.253 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:34 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:00:34] ENGINE Bus STARTED 2024-09-16T09:00:34.253 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:00:34 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:00:34.252Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003852301s 2024-09-16T09:00:34.835 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:34 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T09:00:34.835 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:34 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T09:00:34.835 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:34 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T09:00:34.835 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:34 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T09:00:34.836 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:34 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T09:00:34.836 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:34 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T09:00:34.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:34 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T09:00:34.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:34 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-16T09:00:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:34 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-16T09:00:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:35 smithi111 ceph-mon[29803]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:00:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:35 smithi111 ceph-mon[40056]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:00:35.906 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:35 smithi114 ceph-mon[33315]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:00:37.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:36 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:37.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:36 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:37.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:36 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:37.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:36 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:37.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:36 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:37.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:36 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:38.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:37 smithi114 ceph-mon[33315]: pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-16T09:00:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:37 smithi114 ceph-mon[33315]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-16T09:00:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:37 smithi114 ceph-mon[33315]: Cluster is now healthy 2024-09-16T09:00:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:37 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:37 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:37 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:00:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:37 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:00:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:37 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[29803]: pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[29803]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[29803]: Cluster is now healthy 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[40056]: pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[40056]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-16T09:00:38.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[40056]: Cluster is now healthy 2024-09-16T09:00:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:00:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:00:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:37 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:00:40.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:39 smithi114 ceph-mon[33315]: pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-16T09:00:40.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:39 smithi111 ceph-mon[29803]: pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-16T09:00:40.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:39 smithi111 ceph-mon[40056]: pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-16T09:00:41.845 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:00:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:00:42.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:41 smithi114 ceph-mon[33315]: pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-09-16T09:00:42.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:41 smithi111 ceph-mon[29803]: pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-09-16T09:00:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:41 smithi111 ceph-mon[40056]: pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-09-16T09:00:43.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:43.675 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:00:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:00:44.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:43 smithi114 ceph-mon[33315]: pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-09-16T09:00:44.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:43 smithi111 ceph-mon[29803]: pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-09-16T09:00:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:43 smithi111 ceph-mon[40056]: pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-09-16T09:00:46.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:45 smithi114 ceph-mon[33315]: pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-09-16T09:00:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:46.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:45 smithi111 ceph-mon[29803]: pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-09-16T09:00:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:46.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:45 smithi111 ceph-mon[40056]: pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-09-16T09:00:46.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:00:48.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:47 smithi114 ceph-mon[33315]: pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 0 objects/s recovering 2024-09-16T09:00:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:47 smithi111 ceph-mon[29803]: pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 0 objects/s recovering 2024-09-16T09:00:48.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:47 smithi111 ceph-mon[40056]: pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 0 objects/s recovering 2024-09-16T09:00:50.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:49 smithi114 ceph-mon[33315]: pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:50.175 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:49 smithi111 ceph-mon[29803]: pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:50.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:49 smithi111 ceph-mon[40056]: pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:51.866 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:00:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:00:52.131 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:51 smithi114 ceph-mon[33315]: pgmap v45: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:51 smithi111 ceph-mon[29803]: pgmap v45: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:52.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:51 smithi111 ceph-mon[40056]: pgmap v45: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:52 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:53.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:52 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:52 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:00:53.675 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:00:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:00:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:00:53.694 INFO:tasks.workunit.client.0.smithi111.stderr:Note: switching to '8293d73f8690540e843a81caec373f9cc29cf705'. 2024-09-16T09:00:53.694 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr: git switch -c 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr:Or undo this operation with: 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr: git switch - 2024-09-16T09:00:53.695 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-09-16T09:00:53.696 INFO:tasks.workunit.client.0.smithi111.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T09:00:53.696 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-09-16T09:00:53.696 INFO:tasks.workunit.client.0.smithi111.stderr:HEAD is now at 8293d73f869 Merge branch 'wip-aclamk-cbt-combined' of https://github.com/aclamk/ceph into wip-guits-main-2024-09-13-1248 2024-09-16T09:00:53.700 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T09:00:53.839 INFO:tasks.workunit.client.0.smithi111.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T09:00:53.841 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T09:00:53.841 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T09:00:54.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:53 smithi114 ceph-mon[33315]: pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:53 smithi111 ceph-mon[29803]: pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:54.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:53 smithi111 ceph-mon[40056]: pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:00:54.562 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T09:00:54.633 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T09:00:54.674 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T09:00:54.676 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T09:00:54.676 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T09:00:54.715 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T09:00:54.718 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-09-16T09:00:54.719 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T09:00:54.735 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-16T09:00:54.736 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-16T09:00:54.737 DEBUG:teuthology.orchestra.run.smithi111:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8293d73f8690540e843a81caec373f9cc29cf705 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-16T09:00:54.806 INFO:tasks.workunit.client.0.smithi111.stderr:+ parallel=1 2024-09-16T09:00:54.806 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' '' = --serial ']' 2024-09-16T09:00:54.806 INFO:tasks.workunit.client.0.smithi111.stderr:+ crimson=0 2024-09-16T09:00:54.806 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' '' = --crimson ']' 2024-09-16T09:00:54.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ color= 2024-09-16T09:00:54.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -t 1 ']' 2024-09-16T09:00:54.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-16T09:00:54.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T09:00:54.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T09:00:54.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ declare -A pids 2024-09-16T09:00:54.808 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.808 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.808 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_aio 2024-09-16T09:00:54.808 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_aio' 2024-09-16T09:00:54.808 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_aio 2024-09-16T09:00:54.808 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.809 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_aio 2024-09-16T09:00:54.810 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76362 2024-09-16T09:00:54.810 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_aio on pid 76362' 2024-09-16T09:00:54.810 INFO:tasks.workunit.client.0.smithi111.stdout:test api_aio on pid 76362 2024-09-16T09:00:54.815 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76362 2024-09-16T09:00:54.815 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.816 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-16T09:00:54.816 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.816 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_aio_pp 2024-09-16T09:00:54.816 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_aio_pp' 2024-09-16T09:00:54.816 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_aio_pp 2024-09-16T09:00:54.816 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.817 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.817 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.817 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.817 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.817 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.817 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.818 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.819 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.819 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.819 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.819 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.819 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.819 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.819 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.819 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.819 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.820 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.820 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.820 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.820 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_aio_pp 2024-09-16T09:00:54.820 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76369 2024-09-16T09:00:54.820 INFO:tasks.workunit.client.0.smithi111.stdout:test api_aio_pp on pid 76369 2024-09-16T09:00:54.825 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_aio_pp on pid 76369' 2024-09-16T09:00:54.825 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76369 2024-09-16T09:00:54.825 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.825 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.825 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_io 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_io' 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_io 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.826 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.828 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.829 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.829 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.829 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.829 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.829 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.829 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_io 2024-09-16T09:00:54.829 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76378 2024-09-16T09:00:54.829 INFO:tasks.workunit.client.0.smithi111.stdout:test api_io on pid 76378 2024-09-16T09:00:54.833 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_io on pid 76378' 2024-09-16T09:00:54.833 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76378 2024-09-16T09:00:54.833 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.833 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.833 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-16T09:00:54.833 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_io_pp 2024-09-16T09:00:54.833 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_io_pp' 2024-09-16T09:00:54.833 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_io_pp 2024-09-16T09:00:54.833 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.834 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.835 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.836 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_io_pp 2024-09-16T09:00:54.837 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76390 2024-09-16T09:00:54.838 INFO:tasks.workunit.client.0.smithi111.stdout:test api_io_pp on pid 76390 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_io_pp on pid 76390' 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76390 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_asio 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_asio' 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_asio 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.843 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.844 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.845 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.846 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.847 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_asio 2024-09-16T09:00:54.849 INFO:tasks.workunit.client.0.smithi111.stdout:test api_asio on pid 76403 2024-09-16T09:00:54.856 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76403 2024-09-16T09:00:54.856 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_asio on pid 76403' 2024-09-16T09:00:54.856 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76403 2024-09-16T09:00:54.856 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.857 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.857 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-16T09:00:54.857 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.857 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.857 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.857 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_list 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_list' 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_list 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.858 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.859 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.860 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.861 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:54.862 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.863 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.864 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.865 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.865 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.865 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.865 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.865 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.865 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.865 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.865 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76362/exe 2024-09-16T09:00:54.865 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_list 2024-09-16T09:00:54.866 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76420 2024-09-16T09:00:54.866 INFO:tasks.workunit.client.0.smithi111.stdout:test api_list on pid 76420 2024-09-16T09:00:54.878 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_list on pid 76420' 2024-09-16T09:00:54.878 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76420 2024-09-16T09:00:54.879 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.879 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.879 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-16T09:00:54.879 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.879 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.879 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.879 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.879 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.879 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_lock 2024-09-16T09:00:54.880 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:54.880 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_lock' 2024-09-16T09:00:54.880 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:54.880 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_lock 2024-09-16T09:00:54.880 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.880 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:54.880 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:54.880 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:54.881 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.882 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.883 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_aio: /' 2024-09-16T09:00:54.884 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.885 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.886 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.887 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:54.888 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:54.889 INFO:tasks.workunit.client.0.smithi111.stdout:test api_lock on pid 76440 2024-09-16T09:00:54.892 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_lock 2024-09-16T09:00:54.892 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76440 2024-09-16T09:00:54.892 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_lock on pid 76440' 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76440 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76369/exe 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_lock_pp 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:54.893 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_lock_pp' 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.894 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:54.895 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:54.897 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_lock_pp 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.898 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.899 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76378/exe 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.900 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-16T09:00:54.901 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-16T09:00:54.902 INFO:tasks.workunit.client.0.smithi111.stdout:test api_lock_pp on pid 76460 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_lock_pp 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76460 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_lock_pp on pid 76460' 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76460 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_misc 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_misc' 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.903 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:54.904 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_io.log 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.905 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.906 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_misc 2024-09-16T09:00:54.907 INFO:tasks.workunit.client.0.smithi111.stdout:test api_misc on pid 76475 2024-09-16T09:00:54.913 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_misc 2024-09-16T09:00:54.913 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76475 2024-09-16T09:00:54.913 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_misc on pid 76475' 2024-09-16T09:00:54.913 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76475 2024-09-16T09:00:54.913 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-16T09:00:54.914 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:54.915 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.916 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.917 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_io: /' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.918 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_misc_pp 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.919 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_misc_pp' 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.920 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76390/exe 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_misc_pp 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.921 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.922 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:54.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.924 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.925 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.926 INFO:tasks.workunit.client.0.smithi111.stdout:test api_misc_pp on pid 76501 2024-09-16T09:00:54.934 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.934 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.934 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.934 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_misc_pp 2024-09-16T09:00:54.934 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.935 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.935 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.935 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.935 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.935 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.935 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.935 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.936 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.936 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:54.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76501 2024-09-16T09:00:54.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_misc_pp on pid 76501' 2024-09-16T09:00:54.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76501 2024-09-16T09:00:54.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.937 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:54.937 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:54.937 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:54.937 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.937 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:54.938 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.938 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:54.938 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:54.938 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.938 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.938 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:54.939 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.939 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:54.939 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.939 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:54.939 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:54.939 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.940 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:54.940 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.940 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:54.940 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.940 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.940 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:54.941 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.941 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.941 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.941 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:54.941 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-16T09:00:54.942 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_tier_pp 2024-09-16T09:00:54.942 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:54.942 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:54.942 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:54.942 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.942 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:54.942 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.943 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:54.943 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:54.943 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.943 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.943 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:54.943 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.943 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:54.944 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.944 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:54.944 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:54.944 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.944 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:54.944 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.945 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:54.945 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:54.945 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_tier_pp' 2024-09-16T09:00:54.945 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:54.945 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:54.946 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:54.946 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.946 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:54.946 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.946 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:54.946 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:54.947 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.947 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.947 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.947 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.947 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:54.947 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.947 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:54.948 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.948 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:54.948 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.948 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.948 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:54.948 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.949 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.949 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.949 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:54.949 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.949 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:54.949 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:54.950 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76420/exe 2024-09-16T09:00:54.950 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76403/exe 2024-09-16T09:00:54.950 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.950 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.950 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.950 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.951 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.951 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_tier_pp 2024-09-16T09:00:54.951 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.951 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.951 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.951 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.952 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76440/exe 2024-09-16T09:00:54.952 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.952 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.952 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.952 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.952 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.953 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:54.953 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.953 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.953 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.953 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.953 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.954 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.954 INFO:tasks.workunit.client.0.smithi111.stdout:test api_tier_pp on pid 76521 2024-09-16T09:00:54.957 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:54.957 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:54.957 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:54.957 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:54.958 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:54.958 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:54.958 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:54.958 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:54.958 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:54.958 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:54.958 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:54.959 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.959 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.959 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.959 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.959 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.959 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.959 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.959 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.960 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.960 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:54.960 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_list: /' 2024-09-16T09:00:54.960 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_tier_pp 2024-09-16T09:00:54.960 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:54.960 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:54.960 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:54.960 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:54.961 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:54.961 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:54.961 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76521 2024-09-16T09:00:54.961 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_tier_pp on pid 76521' 2024-09-16T09:00:54.961 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76521 2024-09-16T09:00:54.961 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:54.961 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.962 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.962 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:54.962 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:54.962 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:54.962 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:54.962 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:54.962 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:54.962 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_list.log 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_lock: /' 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_pool 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_pool' 2024-09-16T09:00:54.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.965 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.965 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.965 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.965 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.965 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.965 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.965 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.965 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_pool 2024-09-16T09:00:54.966 INFO:tasks.workunit.client.0.smithi111.stdout:test api_pool on pid 76536 2024-09-16T09:00:54.968 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_pool 2024-09-16T09:00:54.968 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76536 2024-09-16T09:00:54.968 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_pool on pid 76536' 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76536 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_snapshots 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_snapshots' 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_snapshots 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.970 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_asio: /' 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.971 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.972 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.973 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.974 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.975 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.976 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.977 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.978 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.979 INFO:tasks.workunit.client.0.smithi111.stdout:test api_snapshots on pid 76566 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_snapshots 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76566 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_snapshots on pid 76566' 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76566 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_snapshots_pp 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_snapshots_pp' 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.985 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.986 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:54.987 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:54.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_snapshots_pp 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:54.990 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.991 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:54.992 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:54.993 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76501/exe 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.994 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:54.995 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:54.996 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:54.997 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_snapshots_pp 2024-09-16T09:00:54.997 INFO:tasks.workunit.client.0.smithi111.stdout:test api_snapshots_pp on pid 76620 2024-09-16T09:00:55.000 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76620 2024-09-16T09:00:55.000 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_snapshots_pp on pid 76620' 2024-09-16T09:00:55.000 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76620 2024-09-16T09:00:55.000 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_stat 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_stat' 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.001 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76460/exe 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.002 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.003 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.004 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.005 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.006 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76475/exe 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.008 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.009 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_stat 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.010 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.011 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.012 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.012 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.012 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-16T09:00:55.012 INFO:tasks.workunit.client.0.smithi111.stdout:test api_stat on pid 76698 2024-09-16T09:00:55.015 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_stat 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76698 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_stat on pid 76698' 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_misc: /' 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76698 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_stat_pp 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.016 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_stat_pp' 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_stat_pp 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.017 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.018 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76521/exe 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.020 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.021 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_stat_pp 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76729 2024-09-16T09:00:55.023 INFO:tasks.workunit.client.0.smithi111.stdout:test api_stat_pp on pid 76729 2024-09-16T09:00:55.028 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_stat_pp on pid 76729' 2024-09-16T09:00:55.028 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76729 2024-09-16T09:00:55.028 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.028 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.028 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-16T09:00:55.028 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.029 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.030 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.030 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.030 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.030 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.030 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.030 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.031 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.033 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.034 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.035 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.036 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.037 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_watch_notify 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_watch_notify' 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76536/exe 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.038 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.039 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_watch_notify 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.040 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.041 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_pool: /' 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.042 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.043 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.043 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.043 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.043 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.043 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.043 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.043 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.043 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.044 INFO:tasks.workunit.client.0.smithi111.stdout:test api_watch_notify on pid 76768 2024-09-16T09:00:55.048 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_watch_notify 2024-09-16T09:00:55.048 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76768 2024-09-16T09:00:55.048 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_watch_notify on pid 76768' 2024-09-16T09:00:55.048 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76768 2024-09-16T09:00:55.048 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.049 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.049 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-16T09:00:55.049 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_watch_notify_pp 2024-09-16T09:00:55.049 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_watch_notify_pp' 2024-09-16T09:00:55.049 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.050 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76566/exe 2024-09-16T09:00:55.050 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.050 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.050 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.050 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.050 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.050 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.051 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.051 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.051 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.051 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.051 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.051 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.052 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.052 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.052 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.052 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.052 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.052 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.053 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.053 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.053 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.053 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.053 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.053 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.054 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.054 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.054 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.054 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.054 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.054 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.055 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.055 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.055 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.055 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_watch_notify_pp 2024-09-16T09:00:55.055 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.056 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.056 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.056 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.056 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.056 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.056 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.057 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.057 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.057 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.057 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.057 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-16T09:00:55.057 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.058 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.058 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.058 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.058 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.058 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.058 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.059 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.059 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.059 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.059 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.059 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.059 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.059 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.060 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.060 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.060 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.060 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.060 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.060 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.061 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.061 INFO:tasks.workunit.client.0.smithi111.stdout:test api_watch_notify_pp on pid 76794 2024-09-16T09:00:55.072 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_watch_notify_pp 2024-09-16T09:00:55.072 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76794 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_watch_notify_pp on pid 76794' 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76794 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.073 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76698/exe 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.074 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_cmd 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_cmd' 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.075 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_stat: /' 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.076 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.077 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.078 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.079 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.080 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.081 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.082 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.084 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.085 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76729/exe 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.087 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.088 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_cmd 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.089 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.090 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.090 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_cmd 2024-09-16T09:00:55.090 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.090 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76855 2024-09-16T09:00:55.090 INFO:tasks.workunit.client.0.smithi111.stdout:test api_cmd on pid 76855 2024-09-16T09:00:55.094 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_cmd on pid 76855' 2024-09-16T09:00:55.094 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76855 2024-09-16T09:00:55.094 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76620/exe 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.095 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.096 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.097 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.098 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_cmd_pp 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_cmd_pp' 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76768/exe 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.100 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_cmd_pp 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-16T09:00:55.101 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.102 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.103 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.103 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.103 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.103 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.103 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.103 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.103 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.103 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.103 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.104 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.104 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.104 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.104 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.104 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.104 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.104 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.104 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.105 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.105 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.105 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.105 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.105 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.105 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.105 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.106 INFO:tasks.workunit.client.0.smithi111.stdout:test api_cmd_pp on pid 76918 2024-09-16T09:00:55.110 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_cmd_pp 2024-09-16T09:00:55.111 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76918 2024-09-16T09:00:55.111 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_cmd_pp on pid 76918' 2024-09-16T09:00:55.111 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76918 2024-09-16T09:00:55.111 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.111 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.111 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-16T09:00:55.111 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.111 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.111 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.112 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.112 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.112 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.112 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.112 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.112 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.112 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.113 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.113 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.113 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.113 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.113 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.113 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.113 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.113 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.114 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.115 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.115 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.115 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_service 2024-09-16T09:00:55.115 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.115 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.115 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.115 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_service' 2024-09-16T09:00:55.115 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.115 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_service 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.116 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.117 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.117 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.117 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.117 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.117 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.117 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.117 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.117 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.118 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.120 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.120 INFO:tasks.workunit.client.0.smithi111.stdout:test api_service on pid 76971 2024-09-16T09:00:55.122 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_service 2024-09-16T09:00:55.122 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76971 2024-09-16T09:00:55.122 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_service on pid 76971' 2024-09-16T09:00:55.122 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76971 2024-09-16T09:00:55.122 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.122 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.122 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.122 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_service_pp 2024-09-16T09:00:55.122 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_service_pp' 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.123 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_service_pp 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.124 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.125 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76855/exe 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76794/exe 2024-09-16T09:00:55.126 INFO:tasks.workunit.client.0.smithi111.stdout:test api_service_pp on pid 76993 2024-09-16T09:00:55.134 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_service_pp 2024-09-16T09:00:55.134 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76993 2024-09-16T09:00:55.134 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_service_pp on pid 76993' 2024-09-16T09:00:55.134 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=76993 2024-09-16T09:00:55.134 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_c_write_operations 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_c_write_operations' 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.135 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.136 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.137 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.137 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.137 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.137 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.137 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.137 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_c_write_operations 2024-09-16T09:00:55.137 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.137 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.137 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.138 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.139 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.140 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.141 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.142 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_cmd: /' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_c_write_operations 2024-09-16T09:00:55.144 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77030 2024-09-16T09:00:55.145 INFO:tasks.workunit.client.0.smithi111.stdout:test api_c_write_operations on pid 77030 2024-09-16T09:00:55.147 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_c_write_operations on pid 77030' 2024-09-16T09:00:55.147 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77030 2024-09-16T09:00:55.147 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s api_c_read_operations 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' api_c_read_operations' 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.148 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.149 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.150 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.151 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo api_c_read_operations 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.152 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.153 INFO:tasks.workunit.client.0.smithi111.stdout:test api_c_read_operations on pid 77066 2024-09-16T09:00:55.154 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.154 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=api_c_read_operations 2024-09-16T09:00:55.154 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.154 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77066 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test api_c_read_operations on pid 77066' 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77066 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.155 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.156 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s list_parallel 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' list_parallel' 2024-09-16T09:00:55.157 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo list_parallel 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.158 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76918/exe 2024-09-16T09:00:55.159 INFO:tasks.workunit.client.0.smithi111.stdout:test list_parallel on pid 77090 2024-09-16T09:00:55.163 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=list_parallel 2024-09-16T09:00:55.163 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77090 2024-09-16T09:00:55.163 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test list_parallel on pid 77090' 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77090 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.164 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.165 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.166 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s open_pools_parallel 2024-09-16T09:00:55.166 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.166 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.166 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.166 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.166 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' open_pools_parallel' 2024-09-16T09:00:55.167 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.167 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.167 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76971/exe 2024-09-16T09:00:55.167 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.167 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.167 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.167 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.167 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.167 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.168 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.168 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.168 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.168 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.168 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-16T09:00:55.168 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.168 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.168 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.168 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.169 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.169 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.169 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo open_pools_parallel 2024-09-16T09:00:55.169 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.169 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.169 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.169 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.169 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.170 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.170 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.170 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.170 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.170 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-16T09:00:55.170 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.170 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.170 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.170 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.171 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.172 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.172 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.172 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.172 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.173 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.173 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-16T09:00:55.173 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.173 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.173 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.173 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.173 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.173 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.173 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.174 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.174 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.174 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_service.log 2024-09-16T09:00:55.174 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.174 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.174 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.174 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_service: /' 2024-09-16T09:00:55.174 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-16T09:00:55.175 INFO:tasks.workunit.client.0.smithi111.stdout:test open_pools_parallel on pid 77123 2024-09-16T09:00:55.177 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=open_pools_parallel 2024-09-16T09:00:55.177 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.177 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.177 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77123 2024-09-16T09:00:55.177 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.177 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test open_pools_parallel on pid 77123' 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77123 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.178 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s delete_pools_parallel 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' delete_pools_parallel' 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.179 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.180 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.181 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo delete_pools_parallel 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.183 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.183 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.183 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.183 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.183 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.183 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.183 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.183 INFO:tasks.workunit.client.0.smithi111.stdout:test delete_pools_parallel on pid 77137 2024-09-16T09:00:55.186 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=delete_pools_parallel 2024-09-16T09:00:55.186 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77137 2024-09-16T09:00:55.186 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test delete_pools_parallel on pid 77137' 2024-09-16T09:00:55.186 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77137 2024-09-16T09:00:55.186 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s cls 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' cls' 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.187 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-16T09:00:55.188 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.189 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo cls 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.190 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.191 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77030/exe 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.192 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.193 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.194 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.195 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.196 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.197 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.198 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.199 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.200 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.200 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.200 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-16T09:00:55.200 INFO:tasks.workunit.client.0.smithi111.stdout:test cls on pid 77177 2024-09-16T09:00:55.202 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=cls 2024-09-16T09:00:55.202 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77177 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test cls on pid 77177' 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77177 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77066/exe 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s cmd 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' cmd' 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.203 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo cmd 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.204 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.205 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.206 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=cmd 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.207 INFO:tasks.workunit.client.0.smithi111.stdout:test cmd on pid 77197 2024-09-16T09:00:55.211 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77197 2024-09-16T09:00:55.211 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.211 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test cmd on pid 77197' 2024-09-16T09:00:55.211 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.211 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77197 2024-09-16T09:00:55.211 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.211 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.211 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.211 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.213 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s handler_error 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.214 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' handler_error' 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/76993/exe 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo handler_error 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.217 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.218 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.219 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=handler_error 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77224 2024-09-16T09:00:55.220 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test handler_error on pid 77224' 2024-09-16T09:00:55.221 INFO:tasks.workunit.client.0.smithi111.stdout:test handler_error on pid 77224 2024-09-16T09:00:55.226 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77224 2024-09-16T09:00:55.226 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.227 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.229 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.230 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.231 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.231 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.231 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.231 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.231 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.231 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.231 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.231 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.231 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s io 2024-09-16T09:00:55.232 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' io' 2024-09-16T09:00:55.233 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.233 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.233 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.233 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.233 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.233 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-16T09:00:55.233 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.234 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.234 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.234 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.234 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.234 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.234 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.234 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.235 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.235 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.235 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.235 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.235 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.235 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.235 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.236 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.236 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.236 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.236 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.236 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.236 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.236 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.237 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.237 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.237 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.237 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.237 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.237 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.238 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.238 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.238 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.238 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.238 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.238 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.239 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.239 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.239 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.239 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.239 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.239 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.239 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.240 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.240 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.240 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.240 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-16T09:00:55.240 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.240 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.241 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.241 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.241 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.241 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.241 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.241 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.242 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.242 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.242 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.242 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.242 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.242 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.243 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.243 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.243 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.243 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.243 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.243 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.243 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.243 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.244 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.244 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.244 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.244 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.244 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.244 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.245 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.245 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.245 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.245 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.245 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.245 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.246 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.246 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.246 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.246 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.246 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.247 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.247 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.247 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.247 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.247 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.247 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.248 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.248 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.248 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.248 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.248 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.248 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.248 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.249 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.249 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.249 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.249 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.249 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.249 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.250 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.250 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.250 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.250 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.250 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.250 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.251 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.251 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77177/exe 2024-09-16T09:00:55.251 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.251 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.251 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.251 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.252 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.252 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.252 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.252 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.252 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.252 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.252 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.253 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.253 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.253 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.253 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.253 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77123/exe 2024-09-16T09:00:55.253 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.253 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.254 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.254 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.254 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.254 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.254 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.254 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.254 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.255 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.255 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.255 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.255 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.255 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_cls 2024-09-16T09:00:55.255 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.255 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.256 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.256 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.256 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.256 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.256 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.256 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.256 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.257 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.257 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77137/exe 2024-09-16T09:00:55.257 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ cls: /' 2024-09-16T09:00:55.257 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-16T09:00:55.257 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-16T09:00:55.257 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_cls.log 2024-09-16T09:00:55.258 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo io 2024-09-16T09:00:55.258 INFO:tasks.workunit.client.0.smithi111.stdout:test io on pid 77276 2024-09-16T09:00:55.260 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=io 2024-09-16T09:00:55.260 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77276 2024-09-16T09:00:55.260 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test io on pid 77276' 2024-09-16T09:00:55.261 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77276 2024-09-16T09:00:55.261 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.261 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.261 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s ec_io 2024-09-16T09:00:55.261 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' ec_io' 2024-09-16T09:00:55.261 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.262 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.262 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.262 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.262 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.262 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.262 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.262 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-16T09:00:55.263 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.263 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.263 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.263 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.263 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.263 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.264 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.264 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.264 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.264 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.264 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.265 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.265 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.265 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.265 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.265 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.265 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.266 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.266 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.266 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.266 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.267 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.267 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.267 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.267 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.267 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.267 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.268 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.268 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.268 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.268 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.268 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo ec_io 2024-09-16T09:00:55.269 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.269 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.269 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.269 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.269 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.269 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.270 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.270 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.270 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.270 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.270 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.270 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.270 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.271 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.271 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.271 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.271 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.271 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.271 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.272 INFO:tasks.workunit.client.0.smithi111.stdout:test ec_io on pid 77295 2024-09-16T09:00:55.280 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=ec_io 2024-09-16T09:00:55.281 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77295 2024-09-16T09:00:55.281 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test ec_io on pid 77295' 2024-09-16T09:00:55.281 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77295 2024-09-16T09:00:55.281 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.281 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.282 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s list 2024-09-16T09:00:55.282 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' list' 2024-09-16T09:00:55.282 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.282 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-16T09:00:55.282 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-16T09:00:55.282 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.282 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.283 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.283 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.283 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.283 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.283 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.283 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.284 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.284 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.284 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.284 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.284 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.285 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.285 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.285 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.285 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.285 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.285 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.286 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.286 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.286 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.286 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.286 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.286 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.287 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.287 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.287 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.287 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.287 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.287 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.288 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.288 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.288 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.288 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.288 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.288 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.289 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.289 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.289 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.289 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-16T09:00:55.290 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-16T09:00:55.290 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.290 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.290 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.290 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.290 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.290 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.290 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.291 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.291 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.291 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo list 2024-09-16T09:00:55.291 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.291 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.291 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.291 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.291 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.292 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77090/exe 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.293 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.294 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.295 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.296 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.297 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77276/exe 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.298 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_io.log 2024-09-16T09:00:55.300 INFO:tasks.workunit.client.0.smithi111.stdout:test list on pid 77374 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=list 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ io: /' 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77374 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test list on pid 77374' 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77374 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ list_parallel: /' 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_io 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.302 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.303 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s ec_list 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.304 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' ec_list' 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.305 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.306 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.307 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.308 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.309 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.310 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.311 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.312 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo ec_list 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.314 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.314 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.314 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.314 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.314 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.314 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.314 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.314 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.314 INFO:tasks.workunit.client.0.smithi111.stdout:test ec_list on pid 77456 2024-09-16T09:00:55.319 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=ec_list 2024-09-16T09:00:55.319 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77456 2024-09-16T09:00:55.319 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test ec_list on pid 77456' 2024-09-16T09:00:55.319 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77456 2024-09-16T09:00:55.319 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.319 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.319 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.319 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.319 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.320 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77224/exe 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.321 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s misc 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' misc' 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.322 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77197/exe 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.323 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_handler_error 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ handler_error: /' 2024-09-16T09:00:55.324 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_cmd 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.325 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.326 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo misc 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.327 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ cmd: /' 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.328 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.329 INFO:tasks.workunit.client.0.smithi111.stdout:test misc on pid 77536 2024-09-16T09:00:55.336 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=misc 2024-09-16T09:00:55.336 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77536 2024-09-16T09:00:55.336 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test misc on pid 77536' 2024-09-16T09:00:55.336 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77536 2024-09-16T09:00:55.336 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.337 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.338 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.339 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s pool 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.340 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' pool' 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77295/exe 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.341 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.342 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ ec_io: /' 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_ec_io 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.343 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.344 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77374/exe 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo pool 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.345 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_list 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ list: /' 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.346 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.347 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_list.log 2024-09-16T09:00:55.347 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=pool 2024-09-16T09:00:55.347 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77630 2024-09-16T09:00:55.347 INFO:tasks.workunit.client.0.smithi111.stdout:test pool on pid 77630 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test pool on pid 77630' 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77630 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.349 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.351 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s read_operations 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' read_operations' 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.352 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo read_operations 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=read_operations 2024-09-16T09:00:55.353 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.354 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.354 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.354 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77678 2024-09-16T09:00:55.354 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test read_operations on pid 77678' 2024-09-16T09:00:55.354 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77678 2024-09-16T09:00:55.354 INFO:tasks.workunit.client.0.smithi111.stdout:test read_operations on pid 77678 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s snapshots 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' snapshots' 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.358 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.359 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo snapshots 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.360 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.361 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.363 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.363 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.363 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.363 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.363 INFO:tasks.workunit.client.0.smithi111.stdout:test snapshots on pid 77697 2024-09-16T09:00:55.365 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.365 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.365 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=snapshots 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.366 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77697 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test snapshots on pid 77697' 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77697 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s watch_notify 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' watch_notify' 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77456/exe 2024-09-16T09:00:55.367 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.368 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.369 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo watch_notify 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.371 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.372 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.373 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.373 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.373 INFO:tasks.workunit.client.0.smithi111.stdout:test watch_notify on pid 77720 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.376 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=watch_notify 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.377 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77720 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test watch_notify on pid 77720' 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77720 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ ec_list: /' 2024-09-16T09:00:55.378 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:++ printf %25s write_operations 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:+ r=' write_operations' 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.379 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77536/exe 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_ec_list 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.380 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.381 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.382 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo write_operations 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.383 INFO:tasks.workunit.client.0.smithi111.stdout:test write_operations on pid 77761 2024-09-16T09:00:55.391 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.391 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.391 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.391 INFO:tasks.workunit.client.0.smithi111.stderr:+ ff=write_operations 2024-09-16T09:00:55.391 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.391 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.391 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.391 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77761 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 'test write_operations on pid 77761' 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ pids[$f]=77761 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ ret=0 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76521 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76521 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_misc 2024-09-16T09:00:55.392 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_misc.log 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ misc: /' 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:+ . /etc/bashrc 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -z '' ']' 2024-09-16T09:00:55.393 INFO:tasks.workunit.client.0.smithi111.stderr:++ BASHRCSOURCED=Y 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ shopt -q login_shell 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:+++ umask 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.394 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ SHELL=/bin/bash 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T09:00:55.395 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.396 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.397 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.399 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.400 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T09:00:55.401 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:+++ return 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export S_COLORS=auto 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:+++ S_COLORS=auto 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.402 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.403 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.404 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77697/exe 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.405 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.406 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.407 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T09:00:55.408 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77678/exe 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77630/exe 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.409 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_snapshots 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.410 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ read_operations: /' 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.411 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_read_operations 2024-09-16T09:00:55.412 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ pool: /' 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_pool.log 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T09:00:55.413 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ snapshots: /' 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.414 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_pool 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/locale 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T09:00:55.415 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T09:00:55.416 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77720/exe 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:++++ LANG=en_US.UTF-8 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++ LANG=en_US.UTF-8 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++ unset LANG_backup config 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n '' ']' 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' dumb = linux ']' 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.417 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/less.sh 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:++ '[' '' ']' 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.418 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+++++ readlink /proc/77761/exe 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ watch_notify: /' 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:++++ basename /usr/bin/bash 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_declare='declare -f' 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+++ which_opt=-f 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export which_declare 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:+++ export -f which 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset i 2024-09-16T09:00:55.419 INFO:tasks.workunit.client.0.smithi111.stderr:++ unset -f pathmunge 2024-09-16T09:00:55.420 INFO:tasks.workunit.client.0.smithi111.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T09:00:55.420 INFO:tasks.workunit.client.0.smithi111.stderr:+ export PATH 2024-09-16T09:00:55.420 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T09:00:55.420 INFO:tasks.workunit.client.0.smithi111.stderr:+ unset rc 2024-09-16T09:00:55.420 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_write_operations 2024-09-16T09:00:55.420 INFO:tasks.workunit.client.0.smithi111.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-16T09:00:55.420 INFO:tasks.workunit.client.0.smithi111.stderr:+ sed 's/^/ write_operations: /' 2024-09-16T09:00:55.420 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph_test_neorados_watch_notify 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2253283378' entity='client.admin' cmd={"prefix":"quorum_status"} : dispatch 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi111-77574-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi111-77574-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi111-77719-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2253283378' entity='client.admin' cmd={"prefix":"quorum_status"} : dispatch 2024-09-16T09:00:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi111-77574-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi111-77574-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi111-77719-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2253283378' entity='client.admin' cmd={"prefix":"quorum_status"} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi111-77574-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi111-77574-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:56.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi111-77719-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:57.293 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [----------] Global test environment set-up. 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (64 ms) 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (8 ms) 2024-09-16T09:00:57.294 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (4 ms) 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3 ms) 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1 ms) 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [----------] 12 tests from AsioRados (88 ms total) 2024-09-16T09:00:57.295 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: 2024-09-16T09:00:57.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [----------] Global test environment tear-down 2024-09-16T09:00:57.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2394 ms total) 2024-09-16T09:00:57.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-16T09:00:57.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:57.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi111-77574-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:57.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi111-77719-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:57.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:57.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:57.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: osdmap e66: 8 total, 8 up, 8 in 2024-09-16T09:00:57.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1279168519' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77101-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1579786082' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi111-76496-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4284945174' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi111-76657-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1911361' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi111-76716-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectssmithi111-77719-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:00:57.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2391715081' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:57.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/489948550' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi111-77200-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/791350826' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi111-76789-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2784459039' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLock_smithi111-76526-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:57.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3063055963' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "TooBigPP_smithi111-76450-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2692270062' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3179270194' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosStat_smithi111-76802-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/487488763' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi111-76895-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77101-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.418 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi111-76496-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.418 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1512412090' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIo_smithi111-76468-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.418 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi111-76657-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.418 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi111-76716-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.418 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosList_smithi111-76517-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.418 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi111-76789-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.418 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2693125625' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "TooBig_smithi111-76430-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.418 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634067584' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi111-76863-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1397023866' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77009-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLock_smithi111-76526-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/150472930' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi111-76847-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosStat_smithi111-76802-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi111-76895-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIo_smithi111-76468-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosList_smithi111-76517-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi111-77200-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.419 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "TooBig_smithi111-76430-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.420 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3063055963' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi111-76450-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.420 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2692270062' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.420 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.420 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77101-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.420 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi111-76496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.420 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi111-76657-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.420 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi111-76716-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.420 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi111-76789-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.421 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634067584' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi111-76863-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.421 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1397023866' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77009-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.421 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi111-76526-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.421 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/150472930' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi111-76847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.421 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi111-76802-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.421 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi111-76895-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.421 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi111-76468-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.422 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.422 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi111-76517-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.422 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi111-77200-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.422 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi111-76430-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.422 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:56 smithi114 ceph-mon[33315]: osdmap e67: 8 total, 8 up, 8 in 2024-09-16T09:00:57.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:57.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi111-77574-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:57.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi111-77719-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:57.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:57.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:57.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: osdmap e66: 8 total, 8 up, 8 in 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1279168519' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77101-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1579786082' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi111-76496-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4284945174' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi111-76657-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1911361' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi111-76716-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectssmithi111-77719-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2391715081' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/489948550' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi111-77200-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/791350826' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi111-76789-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2784459039' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLock_smithi111-76526-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:57.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3063055963' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "TooBigPP_smithi111-76450-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2692270062' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3179270194' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosStat_smithi111-76802-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/487488763' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi111-76895-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77101-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi111-76496-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1512412090' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIo_smithi111-76468-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi111-76657-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi111-76716-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosList_smithi111-76517-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi111-76789-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2693125625' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "TooBig_smithi111-76430-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634067584' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi111-76863-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi111-77574-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi111-77719-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: osdmap e66: 8 total, 8 up, 8 in 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1279168519' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77101-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1579786082' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi111-76496-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4284945174' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi111-76657-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1911361' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi111-76716-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectssmithi111-77719-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2391715081' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/489948550' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi111-77200-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/791350826' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi111-76789-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2784459039' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLock_smithi111-76526-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3063055963' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "TooBigPP_smithi111-76450-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2692270062' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3179270194' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosStat_smithi111-76802-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/487488763' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi111-76895-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77101-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi111-76496-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1512412090' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIo_smithi111-76468-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi111-76657-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi111-76716-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosList_smithi111-76517-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi111-76789-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2693125625' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "TooBig_smithi111-76430-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634067584' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi111-76863-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1397023866' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77009-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLock_smithi111-76526-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/150472930' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi111-76847-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosStat_smithi111-76802-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi111-76895-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIo_smithi111-76468-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosList_smithi111-76517-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi111-77200-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "TooBig_smithi111-76430-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3063055963' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi111-76450-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2692270062' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77101-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi111-76496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi111-76657-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi111-76716-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi111-76789-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1397023866' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77009-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosLock_smithi111-76526-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/150472930' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi111-76847-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosStat_smithi111-76802-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi111-76895-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIo_smithi111-76468-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosList_smithi111-76517-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi111-77200-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "TooBig_smithi111-76430-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3063055963' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi111-76450-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2692270062' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77101-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi111-76496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi111-76657-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi111-76716-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi111-76789-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634067584' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi111-76863-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1397023866' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77009-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi111-76526-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/150472930' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi111-76847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi111-76802-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi111-76895-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi111-76468-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi111-76517-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi111-77200-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi111-76430-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[40056]: osdmap e67: 8 total, 8 up, 8 in 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634067584' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi111-76863-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1397023866' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77009-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi111-76526-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.436 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/150472930' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi111-76847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi111-76802-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi111-76895-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi111-76468-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi111-76517-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi111-77200-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi111-76430-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:57.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:56 smithi111 ceph-mon[29803]: osdmap e67: 8 total, 8 up, 8 in 2024-09-16T09:00:57.596 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [----------] Global test environment set-up. 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosList.ListObjects (415 ms) 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (34 ms) 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo1 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo2 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo3 2024-09-16T09:00:57.597 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo4 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo5 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo1 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: myset foo6,foo7 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo7 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo6 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns1:foo4 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns1:foo5 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns2:foo7 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns2:foo6 2024-09-16T09:00:57.598 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns1:foo1 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: :foo1 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: :foo2 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: :foo3 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (89 ms) 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 1 0 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 10 0 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 13 0 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 7 0 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 14 0 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 0 0 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 15 0 2024-09-16T09:00:57.599 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 11 0 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 5 0 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 8 0 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 6 0 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 3 0 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 4 0 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 12 0 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 9 0 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 2 0 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (69 ms) 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-16T09:00:57.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: x cursor=MIN 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=1 cursor=13:02547ec2:::1:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=10 cursor=13:52ea6a34:::10:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=13 cursor=13:566253c9:::13:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=7 cursor=13:5c6b0b28:::7:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=14 cursor=13:62a1935d:::14:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=0 cursor=13:6cac518f:::0:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=15 cursor=13:863748b0:::15:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=11 cursor=13:89d3ae78:::11:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=5 cursor=13:b29083e3:::5:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=8 cursor=13:bd63b0f1:::8:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=6 cursor=13:c4fdafeb:::6:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=3 cursor=13:cfc208b3:::3:head 2024-09-16T09:00:57.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=4 cursor=13:d83876eb:::4:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=12 cursor=13:de5d7c5f:::12:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=9 cursor=13:e960b815:::9:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > oid=2 cursor=13:f905c69b:::2:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=1 cursor=13:02547ec2:::1:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=10 cursor=13:52ea6a34:::10:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=13 cursor=13:566253c9:::13:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:566253c9:::13:head 2024-09-16T09:00:57.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-09-16T09:00:57.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=7 cursor=13:5c6b0b28:::7:head 2024-09-16T09:00:57.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-09-16T09:00:57.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-09-16T09:00:57.676 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=14 cursor=13:62a1935d:::14:head 2024-09-16T09:00:57.676 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-09-16T09:00:57.676 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-09-16T09:00:57.676 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=0 cursor=13:6cac518f:::0:head 2024-09-16T09:00:57.676 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-09-16T09:00:57.676 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=15 cursor=13:863748b0:::15:head 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:863748b0:::15:head 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=11 cursor=13:89d3ae78:::11:head 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=5 cursor=13:b29083e3:::5:head 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=8 cursor=13:bd63b0f1:::8:head 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-09-16T09:00:57.677 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=6 cursor=13:c4fdafeb:::6:head 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=3 cursor=13:cfc208b3:::3:head 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=4 cursor=13:d83876eb:::4:head 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=12 cursor=13:de5d7c5f:::12:head 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=9 cursor=13:e960b815:::9:head 2024-09-16T09:00:57.678 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:e960b815:::9:head 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : oid=2 cursor=13:f905c69b:::2:head 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:62a1935d:::14:head expected=13:62a1935d:::14:head 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=14 expected=14 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:5c6b0b28:::7:head expected=13:5c6b0b28:::7:head 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-09-16T09:00:57.679 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=7 expected=7 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:02547ec2:::1:head expected=13:02547ec2:::1:head 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=1 expected=1 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:52ea6a34:::10:head expected=13:52ea6a34:::10:head 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=10 expected=10 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:f905c69b:::2:head expected=13:f905c69b:::2:head 2024-09-16T09:00:57.680 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=2 expected=2 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:e960b815:::9:head 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:e960b815:::9:head expected=13:e960b815:::9:head 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=9 expected=9 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:c4fdafeb:::6:head expected=13:c4fdafeb:::6:head 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=6 expected=6 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:566253c9:::13:head 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:566253c9:::13:head expected=13:566253c9:::13:head 2024-09-16T09:00:57.681 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-09-16T09:00:57.839 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : ent list_parallel: process_1_[77454]: starting. 2024-09-16T09:00:57.839 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_1_[77454]: creating pool ceph_test_rados_list_parallel.smithi111-77347 2024-09-16T09:00:57.839 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_1_[77454]: created object 0... 2024-09-16T09:00:57.839 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_1_[77454]: created object 25... 2024-09-16T09:00:57.839 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_1_[77454]: created object 49... 2024-09-16T09:00:57.839 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_1_[77454]: finishing. 2024-09-16T09:00:57.839 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_1_[77454]: shutting down. 2024-09-16T09:00:57.839 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_2_[77455]: starting. 2024-09-16T09:00:57.839 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_2_[77455]: listing objects. 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_2_[77455]: listed object 0... 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_2_[77455]: listed object 25... 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_2_[77455]: saw 50 objects 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_2_[77455]: shutting down. 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_3_[78034]: starting. 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_3_[78034]: creating pool ceph_test_rados_list_parallel.smithi111-77347 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_3_[78034]: created object 0... 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_3_[78034]: created object 25... 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_3_[78034]: created object 49... 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_3_[78034]: finishing. 2024-09-16T09:00:57.840 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_3_[78034]: shutting down. 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_4_[78035]: starting. 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_4_[78035]: listing objects. 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_4_[78035]: listed object 0... 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_4_[78035]: listed object 25... 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_4_[78035]: saw 43 objects 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_4_[78035]: shutting down. 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_5_[78036]: starting. 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_5_[78036]: removed 25 objects... 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_5_[78036]: removed half of the objects 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_5_[78036]: removed 50 objects... 2024-09-16T09:00:57.841 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_5_[78036]: removed 50 objects 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_5_[78036]: shutting down. 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_6_[78082]: starting. 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_6_[78082]: creating pool ceph_test_rados_list_parallel.smithi111-77347 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_6_[78082]: created object 0... 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_6_[78082]: created object 25... 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_6_[78082]: created object 49... 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_6_[78082]: finishing. 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_6_[78082]: shutting down. 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_7_[78083]: starting. 2024-09-16T09:00:57.842 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_7_[78083]: listing objects. 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_7_[78083]: listed object 0... 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_7_[78083]: listed object 25... 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_7_[78083]: listed object 50... 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_7_[78083]: saw 56 objects 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_7_[78083]: shutting down. 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_8_[78084]: starting. 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_8_[78084]: added 25 objects... 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_8_[78084]: added half of the objects 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_8_[78084]: added 50 objects... 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_8_[78084]: added 50 objects 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_8_[78084]: shutting down. 2024-09-16T09:00:57.843 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.844 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:57.910 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ********************** api_cmd_pp: Running main() from gmock_main.cc 2024-09-16T09:00:57.910 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-16T09:00:57.910 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-16T09:00:57.910 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-16T09:00:57.910 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-16T09:00:57.910 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (69 ms) 2024-09-16T09:00:57.910 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-16T09:00:57.911 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (62 ms) 2024-09-16T09:00:57.911 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-16T09:00:57.911 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2711 ms) 2024-09-16T09:00:57.911 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2842 ms total) 2024-09-16T09:00:57.911 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: 2024-09-16T09:00:57.911 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-16T09:00:57.911 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2843 ms total) 2024-09-16T09:00:57.911 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-16T09:00:57.917 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: Running main() from gmock_main.cc 2024-09-16T09:00:57.917 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-16T09:00:57.917 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: [----------] Global test environment set-up. 2024-09-16T09:00:57.917 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-16T09:00:57.917 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-16T09:00:57.918 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2708 ms) 2024-09-16T09:00:57.918 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: [----------] 1 test from neocls_handler_error (2708 ms total) 2024-09-16T09:00:57.918 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: 2024-09-16T09:00:57.918 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: [----------] Global test environment tear-down 2024-09-16T09:00:57.918 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2708 ms total) 2024-09-16T09:00:57.918 INFO:tasks.workunit.client.0.smithi111.stdout: handler_error: [ PASSED ] 1 test. 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: Running main() from gmock_main.cc 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: [----------] Global test environment set-up. 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: [----------] 1 test from NeoRadosCls 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: [ OK ] NeoRadosCls.DNE (2798 ms) 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: [----------] 1 test from NeoRadosCls (2798 ms total) 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: [----------] Global test environment tear-down 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: [==========] 1 test from 1 test suite ran. (2798 ms total) 2024-09-16T09:00:57.921 INFO:tasks.workunit.client.0.smithi111.stdout: cls: [ PASSED ] 1 test. 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (373 ms) 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (12 ms) 2024-09-16T09:00:57.942 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (15 ms) 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (28 ms) 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (8 ms) 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (6 ms) 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (6 ms) 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (7 ms) 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (7 ms) 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-16T09:00:57.943 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (5 ms) 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (13 ms) 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (9 ms) 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (9 ms) 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-09-16T09:00:57.944 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (512 ms total) 2024-09-16T09:00:57.945 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: 2024-09-16T09:00:57.945 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-16T09:00:57.945 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2845 ms total) 2024-09-16T09:00:57.945 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-16T09:00:58.325 INFO:tasks.workunit.client.0.smithi111.stdout:********* 2024-09-16T09:00:58.325 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_9_[78130]: starting. 2024-09-16T09:00:58.325 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_9_[78130]: creating pool ceph_test_rados_list_parallel.smithi111-77347 2024-09-16T09:00:58.325 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_9_[78130]: created object 0... 2024-09-16T09:00:58.325 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_9_[78130]: created object 25... 2024-09-16T09:00:58.325 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_9_[78130]: created object 49... 2024-09-16T09:00:58.325 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_9_[78130]: finishing. 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_9_[78130]: shutting down. 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_10_[78131]: starting. 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_10_[78131]: listing objects. 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_10_[78131]: listed object 0... 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_10_[78131]: listed object 25... 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_10_[78131]: listed object 50... 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_10_[78131]: listed object 75... 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_10_[78131]: saw 98 objects 2024-09-16T09:00:58.326 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_10_[78131]: shutting down. 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_13_[78134]: starting. 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_13_[78134]: removed 25 objects... 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_13_[78134]: removed half of the objects 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_13_[78134]: removed 50 objects... 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_13_[78134]: removed 50 objects 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_13_[78134]: shutting down. 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.327 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_11_[78132]: starting. 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_11_[78132]: added 25 objects... 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_11_[78132]: added half of the objects 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_11_[78132]: added 50 objects... 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_11_[78132]: added 50 objects 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_11_[78132]: shutting down. 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_12_[78133]: starting. 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_12_[78133]: added 25 objects... 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_12_[78133]: added half of the objects 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_12_[78133]: added 50 objects... 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_12_[78133]: added 50 objects 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_12_[78133]: shutting down. 2024-09-16T09:00:58.328 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_14_[78512]: starting. 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_14_[78512]: creating pool ceph_test_rados_list_parallel.smithi111-77347 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_14_[78512]: created object 0... 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_14_[78512]: created object 25... 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_14_[78512]: created object 49... 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_14_[78512]: finishing. 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_14_[78512]: shutting down. 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.329 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: starting. 2024-09-16T09:00:58.330 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: listing objects. 2024-09-16T09:00:58.330 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: listed object 0... 2024-09-16T09:00:58.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:57 smithi114 ceph-mon[33315]: pgmap v49: 1220 pgs: 1088 creating+peering, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:00:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:57 smithi114 ceph-mon[33315]: Health check failed: 34 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:00:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:00:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi111-77719-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:00:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:00:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:00:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:57 smithi114 ceph-mon[33315]: osdmap e68: 8 total, 8 up, 8 in 2024-09-16T09:00:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[40056]: pgmap v49: 1220 pgs: 1088 creating+peering, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:00:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[40056]: Health check failed: 34 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:00:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:00:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi111-77719-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:00:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:00:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:00:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[40056]: osdmap e68: 8 total, 8 up, 8 in 2024-09-16T09:00:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[29803]: pgmap v49: 1220 pgs: 1088 creating+peering, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:00:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[29803]: Health check failed: 34 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:00:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:00:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi111-77719-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:00:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:00:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:00:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[29803]: osdmap e68: 8 total, 8 up, 8 in 2024-09-16T09:00:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:58.917 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: listed object 25... 2024-09-16T09:00:58.917 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: listed object 50... 2024-09-16T09:00:58.917 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: listed object 75... 2024-09-16T09:00:58.917 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: listed object 100... 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: listed object 125... 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: saw 150 objects 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_15_[78513]: shutting down. 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_16_[78514]: starting. 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_16_[78514]: added 25 objects... 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_16_[78514]: added half of the objects 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_16_[78514]: added 50 objects... 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_16_[78514]: added 50 objects 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: process_16_[78514]: shutting down. 2024-09-16T09:00:58.918 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.919 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.919 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.919 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.919 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******************************* 2024-09-16T09:00:58.919 INFO:tasks.workunit.client.0.smithi111.stdout: list_parallel: ******* SUCCESS ********** 2024-09-16T09:00:58.994 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-16T09:00:58.994 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-16T09:00:58.994 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-16T09:00:58.994 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-16T09:00:58.994 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-16T09:00:58.995 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (57 ms) 2024-09-16T09:00:58.995 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-16T09:00:58.995 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (81 ms) 2024-09-16T09:00:58.995 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-16T09:00:58.995 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2752 ms) 2024-09-16T09:00:58.995 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-16T09:00:58.995 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:56.940600+0000 mon.a [WRN] Health check failed: 34 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:00:58.995 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:57.895951+0000 mon.a [INF] from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi111-77719-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:00:58.995 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:57.896037+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi111-77574-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:00:58.996 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:57.896102+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi111-77003-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:00:58.996 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:57.922200+0000 mon.c [INF] from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:58.996 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:57.926971+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:58.996 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.046305+0000 mon.c [INF] from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:58.996 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.053714+0000 mon.c [INF] from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:58.996 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.083646+0000 mon.c [INF] from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:58.997 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.085547+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:58.997 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.087308+0000 mon.c [INF] from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:58.997 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.099581+0000 mon.c [INF] from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:58.997 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.108321+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:58.997 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.116232+0000 mon.c [INF] from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:58.998 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.120695+0000 mon.b [INF] from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T09:00:58.998 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.123585+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:58.998 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.125693+0000 client.admin [INF] onexx 2024-09-16T09:00:58.998 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.126649+0000 mon.c [INF] from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:58.998 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.137010+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T09:00:58.998 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.137240+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:58.999 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.137463+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:58.999 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.137674+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:58.999 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.137893+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:58.999 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.142834+0000 mon.c [INF] from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:58.999 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.143594+0000 mon.c [INF] from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.000 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.143852+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.000 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.143978+0000 mon.c [INF] from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.000 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.144408+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.000 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.144812+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.000 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.146680+0000 mon.c [INF] from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.001 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.147374+0000 mon.c [INF] from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.001 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.147693+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.001 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.148069+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.001 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.902071+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:59.001 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.902161+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.002 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.902242+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.002 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.902309+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.002 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.902380+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.002 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.909732+0000 mon.b [INF] from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:59.002 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.910695+0000 mon.c [INF] from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.002 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.910904+0000 mon.c [INF] from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.002 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.911144+0000 mon.c [INF] from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.002 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.911311+0000 mon.c [INF] from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T09:00:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: onexx 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: osdmap e69: 8 total, 8 up, 8 in 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2064602573' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1141749187' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2877065268' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWrite_smithi111-76430-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: onexx 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: onexx 2024-09-16T09:00:59.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-16T09:00:59.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: osdmap e69: 8 total, 8 up, 8 in 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2064602573' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1141749187' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2877065268' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWrite_smithi111-76430-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi111-76468-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi111-76802-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: osdmap e69: 8 total, 8 up, 8 in 2024-09-16T09:00:59.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2064602573' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1141749187' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2877065268' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWrite_smithi111-76430-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:00:59.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:00.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: pgmap v52: 636 pgs: 152 unknown, 352 creating+peering, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:01:00.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-16T09:01:00.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["twoxx"]} : dispatch 2024-09-16T09:01:00.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: twoxx 2024-09-16T09:01:00.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["twoxx"]} : dispatch 2024-09-16T09:01:00.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:00.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.0"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.1"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.2"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.3"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.4"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.5"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.6"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.7"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.8"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.9"} : dispatch 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1141749187' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2877065268' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi111-76430-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: osdmap e70: 8 total, 8 up, 8 in 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:00:59 smithi111 ceph-mon[29803]: pgmap v52: 636 pgs: 152 unknown, 352 creating+peering, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:01:00.330 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["twoxx"]} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: twoxx 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["twoxx"]} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.0"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.1"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.2"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.3"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.4"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.5"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.6"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.7"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.8"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.9"} : dispatch 2024-09-16T09:01:00.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1141749187' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2877065268' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi111-76430-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: osdmap e70: 8 total, 8 up, 8 in 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:00.332 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: pgmap v52: 636 pgs: 152 unknown, 352 creating+peering, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:01:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-16T09:01:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["twoxx"]} : dispatch 2024-09-16T09:01:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: twoxx 2024-09-16T09:01:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["twoxx"]} : dispatch 2024-09-16T09:01:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.0"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.1"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.2"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.3"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.4"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.5"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.6"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.7"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.8"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.9"} : dispatch 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1141749187' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2877065268' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi111-76430-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:00.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/753216297' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:01:00.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: osdmap e70: 8 total, 8 up, 8 in 2024-09-16T09:01:00.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:00.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:00.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:01:00.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:01:00.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:00.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:00:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:00.932 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.919006+0000 mon.c [INF] from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true open_pools_parallel: process_1_[77332]: starting. 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_1_[77332]: creating pool ceph_test_rados_open_pools_parallel.smithi111-77273 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_1_[77332]: created object 0... 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_1_[77332]: created object 25... 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_1_[77332]: created object 49... 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_1_[77332]: finishing. 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_1_[77332]: shutting down. 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_2_[77335]: starting. 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_2_[77335]: rados_pool_create. 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_2_[77335]: rados_ioctx_create. 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_2_[77335]: shutting down. 2024-09-16T09:01:00.933 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: ******************************* 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_3_[78210]: starting. 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_3_[78210]: creating pool ceph_test_rados_open_pools_parallel.smithi111-77273 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_3_[78210]: created object 0... 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_3_[78210]: created object 25... 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_3_[78210]: created object 49... 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_3_[78210]: finishing. 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_3_[78210]: shutting down. 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: ******************************* 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_4_[78211]: starting. 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_4_[78211]: rados_pool_create. 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_4_[78211]: rados_ioctx_create. 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: process_4_[78211]: shutting down. 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: ******************************* 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: ******************************* 2024-09-16T09:01:00.934 INFO:tasks.workunit.client.0.smithi111.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_1_[77395]: starting. 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_1_[77395]: creating pool ceph_test_rados_delete_pools_parallel.smithi111-77286 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_1_[77395]: created object 0... 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_1_[77395]: created object 25... 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_1_[77395]: created object 49... 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_1_[77395]: finishing. 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_1_[77395]: shutting down. 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_2_[77396]: starting. 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_2_[77396]: deleting pool ceph_test_rados_delete_pools_parallel.smithi111-77286 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_2_[77396]: shutting down. 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: ******************************* 2024-09-16T09:01:00.948 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_3_[78215]: starting. 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_3_[78215]: creating pool ceph_test_rados_delete_pools_parallel.smithi111-77286 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_3_[78215]: created object 0... 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_3_[78215]: created object 25... 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_3_[78215]: created object 49... 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_3_[78215]: finishing. 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_3_[78215]: shutting down. 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: ******************************* 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_5_[78217]: starting. 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_5_[78217]: listing objects. 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_5_[78217]: listed object 0... 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_5_[78217]: listed object 25... 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_5_[78217]: saw 50 objects 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_5_[78217]: shutting down. 2024-09-16T09:01:00.949 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: ******************************* 2024-09-16T09:01:00.950 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_4_[78216]: starting. 2024-09-16T09:01:00.950 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_4_[78216]: deleting pool ceph_test_rados_delete_pools_parallel.smithi111-77286 2024-09-16T09:01:00.950 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: process_4_[78216]: shutting down. 2024-09-16T09:01:00.950 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: ******************************* 2024-09-16T09:01:00.950 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: ******************************* 2024-09-16T09:01:00.950 INFO:tasks.workunit.client.0.smithi111.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-16T09:01:01.335 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-16T09:01:01.336 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: 15.4 deep-scrub starts 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: 15.4 deep-scrub ok 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.337 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: 15.0 deep-scrub starts 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: 15.0 deep-scrub ok 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/353948289' entity='client.admin' cmd={"prefix":"status"} : dispatch 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: 15.1 deep-scrub starts 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: 15.1 deep-scrub ok 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:01:01.338 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: osdmap e71: 8 total, 8 up, 8 in 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:01.339 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.340 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.340 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:01.340 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-09-16T09:01:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-09-16T09:01:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-09-16T09:01:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-09-16T09:01:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: 15.4 deep-scrub starts 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: 15.4 deep-scrub ok 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-09-16T09:01:01.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: 15.4 deep-scrub starts 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: 15.4 deep-scrub ok 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: 15.0 deep-scrub starts 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: 15.0 deep-scrub ok 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/353948289' entity='client.admin' cmd={"prefix":"status"} : dispatch 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: 15.1 deep-scrub starts 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: 15.1 deep-scrub ok 2024-09-16T09:01:01.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:01.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: osdmap e71: 8 total, 8 up, 8 in 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: 15.0 deep-scrub starts 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: 15.0 deep-scrub ok 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/353948289' entity='client.admin' cmd={"prefix":"status"} : dispatch 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: 15.1 deep-scrub starts 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: 15.1 deep-scrub ok 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:01.432 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: osdmap e71: 8 total, 8 up, 8 in 2024-09-16T09:01:01.433 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:01.434 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:01:01.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:01.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:01.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:01.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:01.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:01.435 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:01.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:01:01 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:01:01.963 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: Running main() from gmock_main.cc 2024-09-16T09:01:01.963 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [----------] Global test environment set-up. 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1726 ms) 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (1976 ms) 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3041 ms) 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6744 ms total) 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [----------] Global test environment tear-down 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6744 ms total) 2024-09-16T09:01:01.964 INFO:tasks.workunit.client.0.smithi111.stdout: cmd: [ PASSED ] 3 tests. 2024-09-16T09:01:02.013 INFO:tasks.workunit.client.0.smithi111.stdout:} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.919277+0000 mon.c [INF] from='client.? 172.21.15.111:0/2064602573' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.927998+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi111-77574-1"} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.928311+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.929069+0000 mon.a [INF] from='client.? 172.21.15.111:0/1141749187' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.929268+0000 mon.a [INF] from='client.? 172.21.15.111:0/2877065268' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWrite_smithi111-76430-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.929570+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.930018+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.930450+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.938440+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.014 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:58.938760+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:00:59.996202+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.022735+0000 mon.c [INF] from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.026917+0000 mon.c [INF] from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.031719+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.032388+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.033220+0000 mon.c [INF] from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.033945+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.037977+0000 mon.c [INF] from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.039017+0000 mon.c [INF] from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.039807+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.040424+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.041212+0000 mon.c [INF] from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.015 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.043127+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.016 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.840490+0000 mon.a [INF] from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:02.016 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911318+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi111-76468-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:02.016 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911399+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi111-76847-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:02.016 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911472+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:02.016 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911540+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi111-76802-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:02.016 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911609+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi111-77574-1"}]': finished 2024-09-16T09:01:02.016 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911664+0000 mon.a [INF] from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:01:02.045 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911710+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name" api_io_pp: Running main() from gmock_main.cc 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: seed 76496 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (361 ms) 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (18 ms) 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (7 ms) 2024-09-16T09:01:02.046 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (323 ms) 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (3 ms) 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (6 ms) 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3 ms) 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5 ms) 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (3 ms) 2024-09-16T09:01:02.047 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2 ms) 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (6 ms) 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (3 ms) 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3 ms) 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (9 ms) 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (4 ms) 2024-09-16T09:01:02.048 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (3 ms) 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (4 ms) 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (3 ms) 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (777 ms total) 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1091 ms) 2024-09-16T09:01:02.049 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-16T09:01:02.050 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (21 ms) 2024-09-16T09:01:02.050 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-16T09:01:02.050 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (8 ms) 2024-09-16T09:01:02.086 INFO:tasks.workunit.client.0.smithi111.stdout: ap: "testprofile-LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:01:02.086 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911777+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:02.086 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911849+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:02.086 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.911911+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi111-76526-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.915510+0000 mon.b [INF] from='client.? 172.21.15.111:0/2686828978' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.917900+0000 mon.c [INF] from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.922106+0000 mon.c [INF] from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.922113+0000 mon.a [INF] from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi111-77719-1"} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.923399+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.923817+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.924402+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.925538+0000 mon.c [INF] from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.926469+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.927760+0000 mon.c [INF] from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:00.939840+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.003388+0000 mon.c [INF] from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.024600+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.915787+0000 mon.a [INF] from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:01:02.087 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.915874+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.915951+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.916031+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.916105+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.921668+0000 mon.c [INF] from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.922020+0000 mon.c [INF] from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.922865+0000 mon.c [INF] from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.943558+0000 mon.b [INF] from='client.? 172.21.15.111:0/2265312168' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.945594+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.946238+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.946537+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-16T09:01:02.088 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.946772+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: pgmap v55: 868 pgs: 512 unknown, 224 creating+peering, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:01:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.3 deep-scrub starts 2024-09-16T09:01:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.3 deep-scrub ok 2024-09-16T09:01:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.6 deep-scrub starts 2024-09-16T09:01:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.6 deep-scrub ok 2024-09-16T09:01:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.7 deep-scrub starts 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.7 deep-scrub ok 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.2 deep-scrub starts 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.2 deep-scrub ok 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.5 deep-scrub starts 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: 15.5 deep-scrub ok 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-16T09:01:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: osdmap e72: 8 total, 8 up, 8 in 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2265312168' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4055573627' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1001970688' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: threexx 2024-09-16T09:01:02.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T09:01:02.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: pgmap v55: 868 pgs: 512 unknown, 224 creating+peering, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:01:02.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.3 deep-scrub starts 2024-09-16T09:01:02.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.3 deep-scrub ok 2024-09-16T09:01:02.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.6 deep-scrub starts 2024-09-16T09:01:02.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.6 deep-scrub ok 2024-09-16T09:01:02.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.7 deep-scrub starts 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.7 deep-scrub ok 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.2 deep-scrub starts 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.2 deep-scrub ok 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.5 deep-scrub starts 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: 15.5 deep-scrub ok 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: osdmap e72: 8 total, 8 up, 8 in 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2265312168' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4055573627' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1001970688' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T09:01:02.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: threexx 2024-09-16T09:01:02.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T09:01:02.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: pgmap v55: 868 pgs: 512 unknown, 224 creating+peering, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:01:02.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.3 deep-scrub starts 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.3 deep-scrub ok 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.6 deep-scrub starts 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.6 deep-scrub ok 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.7 deep-scrub starts 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.7 deep-scrub ok 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.2 deep-scrub starts 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.2 deep-scrub ok 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.5 deep-scrub starts 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: 15.5 deep-scrub ok 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:02.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/455051980' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi111-77719-1"}]': finished 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi111-77003-1"}]': finished 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi111-76450-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi111-77574-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3449501201' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: osdmap e72: 8 total, 8 up, 8 in 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2265312168' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:02.431 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4055573627' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1001970688' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: threexx 2024-09-16T09:01:02.432 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T09:01:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: 15.8 deep-scrub starts 2024-09-16T09:01:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: 15.8 deep-scrub ok 2024-09-16T09:01:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-09-16T09:01:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["fourxx"]} : dispatch 2024-09-16T09:01:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi111-76496-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-16T09:01:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: fourxx 2024-09-16T09:01:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["fourxx"]} : dispatch 2024-09-16T09:01:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi111-76496-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2453930823' entity='client.admin' cmd={"prefix":"status"} : dispatch 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: 15.9 deep-scrub starts 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: 15.9 deep-scrub ok 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"}]': finished 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"}]': finished 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:03.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi111-76496-24", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi111-77719-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-4-cache", "mode": "writeback"} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: osdmap e73: 8 total, 8 up, 8 in 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi111-77719-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-4-cache", "mode": "writeback"} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:03.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:03.425 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:01:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: 15.8 deep-scrub starts 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: 15.8 deep-scrub ok 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["fourxx"]} : dispatch 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi111-76496-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: fourxx 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["fourxx"]} : dispatch 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi111-76496-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-16T09:01:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2453930823' entity='client.admin' cmd={"prefix":"status"} : dispatch 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: 15.9 deep-scrub starts 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: 15.9 deep-scrub ok 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"}]': finished 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"}]': finished 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi111-76496-24", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: 15.8 deep-scrub starts 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: 15.8 deep-scrub ok 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["fourxx"]} : dispatch 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi111-76496-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: fourxx 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["fourxx"]} : dispatch 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi111-76496-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2453930823' entity='client.admin' cmd={"prefix":"status"} : dispatch 2024-09-16T09:01:03.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: 15.9 deep-scrub starts 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: 15.9 deep-scrub ok 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi111-76657-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi111-76657-10"}]': finished 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi111-76526-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi111-76526-10"}]': finished 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-4", "overlaypool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi111-76450-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:03.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi111-76496-24", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi111-77719-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-4-cache", "mode": "writeback"} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: osdmap e73: 8 total, 8 up, 8 in 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi111-77719-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-4-cache", "mode": "writeback"} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:03.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi111-77719-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-4-cache", "mode": "writeback"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: osdmap e73: 8 total, 8 up, 8 in 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi111-77719-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-4-cache", "mode": "writeback"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:03.431 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:04.204 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.947758+0000 mon.c [INF] from='client.? 172 watch_notify: Running main() from gmock_main.cc 2024-09-16T09:01:04.205 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-16T09:01:04.205 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-16T09:01:04.205 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-16T09:01:04.205 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-16T09:01:04.205 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: handle_notify cookie 94688541177216 notify_id 287762808833 notifier_gid 24839 2024-09-16T09:01:04.205 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2614 ms) 2024-09-16T09:01:04.205 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-16T09:01:04.206 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: Trying... 2024-09-16T09:01:04.206 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: handle_notify cookie 94688553880800 notify_id 300647710720 notifier_gid 24898 2024-09-16T09:01:04.206 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: Waiting for 3.000000000s 2024-09-16T09:01:04.206 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: Timed out. 2024-09-16T09:01:04.206 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: Flushing... 2024-09-16T09:01:04.206 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: Flushed... 2024-09-16T09:01:04.206 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (6290 ms) 2024-09-16T09:01:04.206 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (8904 ms total) 2024-09-16T09:01:04.207 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: 2024-09-16T09:01:04.207 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-16T09:01:04.207 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (8905 ms total) 2024-09-16T09:01:04.207 INFO:tasks.workunit.client.0.smithi111.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-16T09:01:04.210 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: Running main() from gmock_main.cc 2024-09-16T09:01:04.210 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-16T09:01:04.210 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [----------] Global test environment set-up. 2024-09-16T09:01:04.210 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-16T09:01:04.210 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-16T09:01:04.210 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (362 ms) 2024-09-16T09:01:04.210 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-16T09:01:04.210 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: no timeout :/ 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: no timeout :/ 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: no timeout :/ 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: no timeout :/ 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: no timeout :/ 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (56 ms) 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (6 ms) 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-16T09:01:04.211 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (6 ms) 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (5 ms) 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6 ms) 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (3 ms) 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-16T09:01:04.212 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.RmXattr (12 ms) 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [----------] 14 tests from LibRadosIo (480 ms total) 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1110 ms) 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (9 ms) 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-16T09:01:04.213 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (5 ms) 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (7 ms) 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (3 ms) 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-16T09:01:04.214 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2024-09-16T09:01:04.215 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-16T09:01:04.215 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-09-16T09:01:04.215 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-16T09:01:04.215 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (7 ms) 2024-09-16T09:01:04.215 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1164 ms total) 2024-09-16T09:01:04.215 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: 2024-09-16T09:01:04.215 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [----------] Global test environment tear-down 2024-09-16T09:01:04.215 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9357 ms total) 2024-09-16T09:01:04.215 INFO:tasks.workunit.client.0.smithi111.stdout: api_io: [ PASSED ] 24 tests. 2024-09-16T09:01:04.216 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-16T09:01:04.216 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-16T09:01:04.216 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-16T09:01:04.216 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-16T09:01:04.216 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: seed 76847 2024-09-16T09:01:04.216 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-16T09:01:04.216 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (251 ms) 2024-09-16T09:01:04.216 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (49 ms) 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (12 ms) 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (27 ms) 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (340 ms total) 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-16T09:01:04.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1100 ms) 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (12 ms) 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (12 ms) 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1124 ms total) 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (9208 ms total) 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: Running main() from gmock_main.cc 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-16T09:01:04.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [----------] Global test environment set-up. 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ OK ] LibRadosStat.Stat (355 ms) 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (4 ms) 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ OK ] LibRadosStat.StatNS (23 ms) 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (5 ms) 2024-09-16T09:01:04.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [----------] 5 tests from LibRadosStat (387 ms total) 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1118 ms) 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (8 ms) 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-16T09:01:04.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-09-16T09:01:04.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1128 ms total) 2024-09-16T09:01:04.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: 2024-09-16T09:01:04.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [----------] Global test environment tear-down 2024-09-16T09:01:04.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (9245 ms total) 2024-09-16T09:01:04.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-16T09:01:04.221 INFO:tasks.workunit.client.0.smithi111.stdout: list: Running main() from gmock_main.cc 2024-09-16T09:01:04.221 INFO:tasks.workunit.client.0.smithi111.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [----------] Global test environment set-up. 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [----------] 3 tests from NeoradosList 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [ OK ] NeoradosList.ListObjects (2690 ms) 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3016 ms) 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3277 ms) 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [----------] 3 tests from NeoradosList (8983 ms total) 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [----------] Global test environment tear-down 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [==========] 3 tests from 1 test suite ran. (8983 ms total) 2024-09-16T09:01:04.222 INFO:tasks.workunit.client.0.smithi111.stdout: list: [ PASSED ] 3 tests. 2024-09-16T09:01:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: pgmap v58: 900 pgs: 128 creating+peering, 288 unknown, 484 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 548 op/s 2024-09-16T09:01:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: Health check update: 17 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-09-16T09:01:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"}]': finished 2024-09-16T09:01:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-4-cache", "mode": "writeback"}]': finished 2024-09-16T09:01:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: osdmap e74: 8 total, 8 up, 8 in 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-77003-6", "pg_num": 4} : dispatch 2024-09-16T09:01:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-4"} : dispatch 2024-09-16T09:01:04.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-4"} : dispatch 2024-09-16T09:01:04.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: pgmap v58: 900 pgs: 128 creating+peering, 288 unknown, 484 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 548 op/s 2024-09-16T09:01:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: Health check update: 17 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-09-16T09:01:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"}]': finished 2024-09-16T09:01:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-4-cache", "mode": "writeback"}]': finished 2024-09-16T09:01:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: osdmap e74: 8 total, 8 up, 8 in 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-77003-6", "pg_num": 4} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-4"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-4"} : dispatch 2024-09-16T09:01:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: pgmap v58: 900 pgs: 128 creating+peering, 288 unknown, 484 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 548 op/s 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: Health check update: 17 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi111-77574-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi111-77574-2"}]': finished 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-4-cache", "mode": "writeback"}]': finished 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:04.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2688166532' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1795148405' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3603107921' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: osdmap e74: 8 total, 8 up, 8 in 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-77003-6", "pg_num": 4} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-4"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-4"} : dispatch 2024-09-16T09:01:04.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:05.098 INFO:tasks.workunit.client.0.smithi111.stdout:.21.15.111:0/4055573627' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.098 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.948432+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.098 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.954534+0000 mon.b [INF] from='client.? 172.21.15.111:0/1001970688' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.098 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.954873+0000 mon.b [INF] from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:05.098 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.958113+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi111-76430-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.098 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:01.958363+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi111-77719-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:05.099 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:02.013109+0000 mon.b [INF] from='client.? 172.21.15.111:0/4112118764' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T09:01:05.099 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:02.013772+0000 client.admin [INF] threexx 2024-09-16T09:01:05.099 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: got: 2024-09-16T09:01:02.018019+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-16T09:01:05.099 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7169 ms) 2024-09-16T09:01:05.099 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (10059 ms total) 2024-09-16T09:01:05.099 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: 2024-09-16T09:01:05.099 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-16T09:01:05.099 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (10059 ms total) 2024-09-16T09:01:05.099 INFO:tasks.workunit.client.0.smithi111.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi111-77719-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi111-77719-2"}]': finished 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-77003-6", "pg_num": 4}]': finished 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-4"}]': finished 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: osdmap e75: 8 total, 8 up, 8 in 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2261367162' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3504958720' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:05.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:05.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/732461535' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip_smithi111-76430-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:05.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi111-77719-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi111-77719-2"}]': finished 2024-09-16T09:01:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-77003-6", "pg_num": 4}]': finished 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-4"}]': finished 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: osdmap e75: 8 total, 8 up, 8 in 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2261367162' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3504958720' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/732461535' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip_smithi111-76430-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi111-77719-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi111-77719-2"}]': finished 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi111-76847-7"}]': finished 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi111-76802-7"}]': finished 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi111-76468-16"}]': finished 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-77003-6", "pg_num": 4}]': finished 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-4"}]': finished 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: osdmap e75: 8 total, 8 up, 8 in 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3808361891' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/85882406' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2261367162' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3504958720' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:05.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:05.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/732461535' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip_smithi111-76430-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"} : dispatch 2024-09-16T09:01:05.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:05.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: pgmap v62: 776 pgs: 32 creating+peering, 292 unknown, 452 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 42 MiB/s wr, 615 op/s 2024-09-16T09:01:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/732461535' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi111-76430-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: osdmap e76: 8 total, 8 up, 8 in 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "overlaypool": "test-rados-api-smithi111-77003-6"} : dispatch 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:06.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: pgmap v62: 776 pgs: 32 creating+peering, 292 unknown, 452 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 42 MiB/s wr, 615 op/s 2024-09-16T09:01:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/732461535' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi111-76430-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: osdmap e76: 8 total, 8 up, 8 in 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "overlaypool": "test-rados-api-smithi111-77003-6"} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: pgmap v62: 776 pgs: 32 creating+peering, 292 unknown, 452 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 42 MiB/s wr, 615 op/s 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/732461535' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi111-76430-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-4", "tierpool": "test-rados-api-smithi111-76716-4-cache"}]': finished 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: osdmap e76: 8 total, 8 up, 8 in 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "overlaypool": "test-rados-api-smithi111-77003-6"} : dispatch 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:06.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:06.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:06.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: Running main() from gmock_main.cc 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [----------] Global test environment set-up. 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5687 ms) 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6320 ms) 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12007 ms total) 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [----------] Global test environment tear-down 2024-09-16T09:01:07.240 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12007 ms total) 2024-09-16T09:01:07.241 INFO:tasks.workunit.client.0.smithi111.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "overlaypool": "test-rados-api-smithi111-77003-6"}]': finished 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi111-77574-2"}]': finished 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: osdmap e77: 8 total, 8 up, 8 in 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-77003-6", "mode": "writeback"} : dispatch 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/779132920' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi111-76789-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "overlaypool": "test-rados-api-smithi111-77003-6"}]': finished 2024-09-16T09:01:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi111-77574-2"}]': finished 2024-09-16T09:01:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: osdmap e77: 8 total, 8 up, 8 in 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-77003-6", "mode": "writeback"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/779132920' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi111-76789-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "overlaypool": "test-rados-api-smithi111-77003-6"}]': finished 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi111-77574-2"}]': finished 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: osdmap e77: 8 total, 8 up, 8 in 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3963780667' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-77003-6", "mode": "writeback"} : dispatch 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReadOpsmithi111-77574-2"} : dispatch 2024-09-16T09:01:07.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:07.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:07.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:07.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:07.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:07.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:07.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/779132920' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi111-76789-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:07.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:07.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:07.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:08.217 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: Running main() from gmock_main.cc 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [----------] Global test environment set-up. 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (304 ms) 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLock.LockShared (33 ms) 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1031 ms) 2024-09-16T09:01:08.218 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1006 ms) 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (45 ms) 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLock.Unlock (20 ms) 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (10 ms) 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (2 ms) 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [----------] 8 tests from LibRadosLock (2451 ms total) 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-16T09:01:08.219 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (388 ms) 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (4 ms) 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1060 ms) 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1017 ms) 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3 ms) 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (4 ms) 2024-09-16T09:01:08.220 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-16T09:01:08.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (2 ms) 2024-09-16T09:01:08.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (2483 ms total) 2024-09-16T09:01:08.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: 2024-09-16T09:01:08.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [----------] Global test environment tear-down 2024-09-16T09:01:08.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13348 ms total) 2024-09-16T09:01:08.221 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-16T09:01:08.221 INFO:tasks.workunit.client.0.smithi111.stdout: pool: Running main() from gmock_main.cc 2024-09-16T09:01:08.221 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-16T09:01:08.221 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [----------] Global test environment set-up. 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ OK ] NeoRadosPools.PoolList (1631 ms) 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2006 ms) 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (1993 ms) 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (3291 ms) 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2012 ms) 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-16T09:01:08.222 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (1989 ms) 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [----------] 6 tests from NeoRadosPools (12922 ms total) 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: pool: 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [----------] Global test environment tear-down 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [==========] 6 tests from 1 test suite ran. (12922 ms total) 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: pool: [ PASSED ] 6 tests. 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: seed 76657 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (294 ms) 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (56 ms) 2024-09-16T09:01:08.223 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1026 ms) 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1033 ms) 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (26 ms) 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (19 ms) 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5 ms) 2024-09-16T09:01:08.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (2 ms) 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2461 ms total) 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (348 ms) 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (44 ms) 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1047 ms) 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1057 ms) 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-16T09:01:08.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (3 ms) 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (7 ms) 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (2513 ms total) 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13286 ms total) 2024-09-16T09:01:08.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-16T09:01:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: pgmap v65: 584 pgs: 64 creating+peering, 520 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-09-16T09:01:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-77003-6", "mode": "writeback"}]': finished 2024-09-16T09:01:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi111-77574-2"}]': finished 2024-09-16T09:01:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"}]': finished 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi111-77719-2"}]': finished 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"}]': finished 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/779132920' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi111-76789-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: osdmap e78: 8 total, 8 up, 8 in 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3223941803' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip2_smithi111-76430-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip2_smithi111-76430-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2742843192' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/68918491' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd dump"} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: pool 'PoolQuotaPP_smithi111-76450-3' is full (reached quota's max_bytes: 4 KiB) 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:01:08.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi111-77719-2"}]': finished 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"}]': finished 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"}]': finished 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi111-76430-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2742843192' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/68918491' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: osdmap e79: 8 total, 8 up, 8 in 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi111-76895-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2"} : dispatch 2024-09-16T09:01:08.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.668 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: pgmap v65: 584 pgs: 64 creating+peering, 520 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-09-16T09:01:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-77003-6", "mode": "writeback"}]': finished 2024-09-16T09:01:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi111-77574-2"}]': finished 2024-09-16T09:01:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"}]': finished 2024-09-16T09:01:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi111-77719-2"}]': finished 2024-09-16T09:01:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"}]': finished 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/779132920' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi111-76789-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: osdmap e78: 8 total, 8 up, 8 in 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3223941803' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip2_smithi111-76430-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip2_smithi111-76430-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2742843192' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/68918491' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: pgmap v65: 584 pgs: 64 creating+peering, 520 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2024-09-16T09:01:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-77003-6", "mode": "writeback"}]': finished 2024-09-16T09:01:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi111-77574-2"}]': finished 2024-09-16T09:01:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi111-76657-10"}]': finished 2024-09-16T09:01:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi111-77719-2"}]': finished 2024-09-16T09:01:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi111-76526-10"}]': finished 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/779132920' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi111-76789-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3528054962' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3182450968' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4051139495' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: osdmap e78: 8 total, 8 up, 8 in 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3223941803' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip2_smithi111-76430-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi111-77719-2"} : dispatch 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"} : dispatch 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"} : dispatch 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip2_smithi111-76430-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2742843192' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/68918491' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.680 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:08.680 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:08.680 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:08.680 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd dump"} : dispatch 2024-09-16T09:01:08.680 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: pool 'PoolQuotaPP_smithi111-76450-3' is full (reached quota's max_bytes: 4 KiB) 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi111-77719-2"}]': finished 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"}]': finished 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"}]': finished 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi111-76430-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2742843192' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.682 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/68918491' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.682 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:08.682 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.682 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: osdmap e79: 8 total, 8 up, 8 in 2024-09-16T09:01:08.682 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi111-76895-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:08.682 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2"} : dispatch 2024-09-16T09:01:08.682 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.682 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd dump"} : dispatch 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: pool 'PoolQuotaPP_smithi111-76450-3' is full (reached quota's max_bytes: 4 KiB) 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:01:08.683 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi111-76496-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi111-77719-2"}]': finished 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi111-76657-10"}]': finished 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi111-76526-10"}]': finished 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi111-76430-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2742843192' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/68918491' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: osdmap e79: 8 total, 8 up, 8 in 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi111-76895-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2"} : dispatch 2024-09-16T09:01:08.684 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:08.685 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi111-77719-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi111-77719-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi111-77719-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsManysmithi111-77719-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"dne","key":"key","value":"value"} : dispatch 2024-09-16T09:01:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:01:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: osdmap e80: 8 total, 8 up, 8 in 2024-09-16T09:01:09.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsManysmithi111-77719-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:09.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:09.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:01:09.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi111-77719-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi111-77719-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi111-77719-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsManysmithi111-77719-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"dne","key":"key","value":"value"} : dispatch 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: osdmap e80: 8 total, 8 up, 8 in 2024-09-16T09:01:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsManysmithi111-77719-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi111-77719-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi111-77719-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi111-76695-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi111-77719-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsManysmithi111-77719-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"dne","key":"key","value":"value"} : dispatch 2024-09-16T09:01:09.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:01:09.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: osdmap e80: 8 total, 8 up, 8 in 2024-09-16T09:01:09.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsManysmithi111-77719-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:09.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:09.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:01:09.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: notify 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1113 ms) 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (4 ms) 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1117 ms total) 2024-09-16T09:01:10.360 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: notify 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (107 ms) 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: notify 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3976 ms) 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (7 ms) 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (6 ms) 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605441248 notify_id 335007449092 notifier_gid 15273 2024-09-16T09:01:10.361 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5 ms) 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605444512 notify_id 335007449093 notifier_gid 15273 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (4 ms) 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605447648 notify_id 335007449088 notifier_gid 15273 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (5 ms) 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605447648 notify_id 335007449094 notifier_gid 15273 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (5 ms) 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605447648 notify_id 335007449095 notifier_gid 15273 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (5 ms) 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-16T09:01:10.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605459696 notify_id 335007449090 notifier_gid 15273 2024-09-16T09:01:10.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (5 ms) 2024-09-16T09:01:10.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-16T09:01:10.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: trying... 2024-09-16T09:01:10.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605447648 notify_id 335007449089 notifier_gid 15273 2024-09-16T09:01:10.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: timed out 2024-09-16T09:01:10.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: flushing 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: pgmap v68: 656 pgs: 232 unknown, 64 creating+peering, 360 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi111-76895-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"}]': finished 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: osdmap e81: 8 total, 8 up, 8 in 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3585457505' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip3_smithi111-76430-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4006883215' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:01:10.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4056563398' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2274630837' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi111-76863-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip3_smithi111-76430-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:10.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: pgmap v68: 656 pgs: 232 unknown, 64 creating+peering, 360 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi111-76895-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"}]': finished 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: osdmap e81: 8 total, 8 up, 8 in 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3585457505' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip3_smithi111-76430-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4006883215' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4056563398' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2274630837' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi111-76863-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip3_smithi111-76430-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: pgmap v68: 656 pgs: 232 unknown, 64 creating+peering, 360 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi111-76895-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"}]': finished 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:01:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: osdmap e81: 8 total, 8 up, 8 in 2024-09-16T09:01:10.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3585457505' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip3_smithi111-76430-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:10.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4006883215' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:01:10.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4056563398' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2274630837' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi111-76863-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTrip3_smithi111-76430-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:10.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:10.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:11.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:11.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi111-77719-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi111-77719-3"}]': finished 2024-09-16T09:01:11.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-16T09:01:11.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4056563398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2274630837' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi111-76863-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi111-76430-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: osdmap e82: 8 total, 8 up, 8 in 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:11.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi111-77719-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi111-77719-3"}]': finished 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4056563398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2274630837' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi111-76863-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi111-76430-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: osdmap e82: 8 total, 8 up, 8 in 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:01:11.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:11.529 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi111-77719-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi111-77719-3"}]': finished 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4056563398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2274630837' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi111-76863-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi111-76430-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: osdmap e82: 8 total, 8 up, 8 in 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1780729112' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:11.530 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1988025402' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:01:11.531 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"} : dispatch 2024-09-16T09:01:11.531 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:01:11.531 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:11.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:01:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:01:12.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_watci_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (2 ms) 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (2 ms) 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (4 ms) 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (4 ms) 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2 ms) 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (4 ms) 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-16T09:01:12.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2 ms) 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (11 ms) 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (6122 ms) 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1086 ms) 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (35 ms) 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-16T09:01:12.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-09-16T09:01:12.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-16T09:01:12.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-09-16T09:01:12.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (8411 ms total) 2024-09-16T09:01:12.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: 2024-09-16T09:01:12.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-16T09:01:12.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (17366 ms total) 2024-09-16T09:01:12.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-09-16T09:01:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:12 smithi114 ceph-mon[33315]: pgmap v71: 688 pgs: 296 unknown, 64 creating+peering, 328 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1"}]': finished 2024-09-16T09:01:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:12 smithi114 ceph-mon[33315]: osdmap e83: 8 total, 8 up, 8 in 2024-09-16T09:01:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[29803]: pgmap v71: 688 pgs: 296 unknown, 64 creating+peering, 328 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1"}]': finished 2024-09-16T09:01:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[29803]: osdmap e83: 8 total, 8 up, 8 in 2024-09-16T09:01:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[40056]: pgmap v71: 688 pgs: 296 unknown, 64 creating+peering, 328 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi111-76496-24"}]': finished 2024-09-16T09:01:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi111-76695-1","app":"app1","key":"key1"}]': finished 2024-09-16T09:01:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[40056]: osdmap e83: 8 total, 8 up, 8 in 2024-09-16T09:01:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"}]': finished 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: osdmap e84: 8 total, 8 up, 8 in 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4057891181' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2751430568' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi111-76430-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi111-76430-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"}]': finished 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: osdmap e84: 8 total, 8 up, 8 in 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4057891181' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2751430568' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi111-76430-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi111-76430-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:13.679 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:01:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi111-76895-12"}]': finished 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: osdmap e84: 8 total, 8 up, 8 in 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi111-76895-12"} : dispatch 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4057891181' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:13.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:13.681 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2751430568' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi111-76430-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:13.681 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi111-76430-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:14.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (332 ms) 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94579842776992 err -107 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (4 ms) 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94579842788960 err -107 2024-09-16T09:01:14.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (46 ms) 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_cb from 24683 notify_id 287762808833 cookie 94579844170784 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (13 ms) 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_cb from 24683 notify_id 287762808832 cookie 94579844207664 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (15 ms) 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_cb from 24683 notify_id 287762808832 cookie 94579844193008 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (7 ms) 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_cb from 24683 notify_id 287762808833 cookie 94579844193008 2024-09-16T09:01:14.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_cb from 24683 notify_id 287762808833 cookie 94579844217280 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (10 ms) 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_cb from 24683 notify_id 287762808834 cookie 94579844193008 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_cb from 24683 notify_id 292057776131 cookie 94579844193008 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3007 ms) 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94579844193008 err -107 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_cb from 24683 notify_id 326417514498 cookie 94579844193008 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5010 ms) 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94579844193008 err -107 2024-09-16T09:01:14.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1007 ms) 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9451 ms total) 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1074 ms) 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1074 ms total) 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-16T09:01:14.262 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (19253 ms total) 2024-09-16T09:01:14.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-16T09:01:14.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: pgmap v74: 560 pgs: 72 creating+peering, 488 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.0 KiB/s wr, 7 op/s 2024-09-16T09:01:14.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:01:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi111-76895-12"}]': finished 2024-09-16T09:01:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4057891181' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-7"}]': finished 2024-09-16T09:01:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi111-76430-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: osdmap e85: 8 total, 8 up, 8 in 2024-09-16T09:01:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: pgmap v74: 560 pgs: 72 creating+peering, 488 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.0 KiB/s wr, 7 op/s 2024-09-16T09:01:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:01:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi111-76895-12"}]': finished 2024-09-16T09:01:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4057891181' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-7"}]': finished 2024-09-16T09:01:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi111-76430-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: osdmap e85: 8 total, 8 up, 8 in 2024-09-16T09:01:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:14.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: pgmap v74: 560 pgs: 72 creating+peering, 488 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.0 KiB/s wr, 7 op/s 2024-09-16T09:01:14.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:14.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:01:14.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3634377453' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi111-76895-12"}]': finished 2024-09-16T09:01:14.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4057891181' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-77162-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:14.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-7"}]': finished 2024-09-16T09:01:14.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi111-76430-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:14.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: osdmap e85: 8 total, 8 up, 8 in 2024-09-16T09:01:14.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:14.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:15.663 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=infra.usagestats t=2024-09-16T09:01:15.272592753Z level=info msg="Usage stats are ready to report" 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: pgmap v77: 680 pgs: 224 unknown, 32 creating+peering, 424 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi111-77719-3"}]': finished 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: osdmap e86: 8 total, 8 up, 8 in 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1742049782' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:15.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: pgmap v77: 680 pgs: 224 unknown, 32 creating+peering, 424 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi111-77719-3"}]': finished 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: osdmap e86: 8 total, 8 up, 8 in 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1742049782' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:15.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: pgmap v77: 680 pgs: 224 unknown, 32 creating+peering, 424 active+clean; 145 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-16T09:01:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi111-77719-3"}]': finished 2024-09-16T09:01:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:01:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: osdmap e86: 8 total, 8 up, 8 in 2024-09-16T09:01:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:15.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3192334516' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:15.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1742049782' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:15.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7"} : dispatch 2024-09-16T09:01:15.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi111-77719-3"} : dispatch 2024-09-16T09:01:16.245 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: Running main() from gmock_main.cc 2024-09-16T09:01:16.245 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-16T09:01:16.245 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [----------] Global test environment set-up. 2024-09-16T09:01:16.245 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-16T09:01:16.245 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6648 ms) 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6283 ms) 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (8039 ms) 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [----------] 3 tests from NeoradosECList (20970 ms total) 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [----------] Global test environment tear-down 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (20970 ms total) 2024-09-16T09:01:16.246 INFO:tasks.workunit.client.0.smithi111.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-16T09:01:16.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1742049782' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7"}]': finished 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi111-77719-3"}]': finished 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: osdmap e87: 8 total, 8 up, 8 in 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3735571060' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RemoveTest_smithi111-76430-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1985010718' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RemoveTest_smithi111-76430-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1742049782' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7"}]': finished 2024-09-16T09:01:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi111-77719-3"}]': finished 2024-09-16T09:01:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: osdmap e87: 8 total, 8 up, 8 in 2024-09-16T09:01:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3735571060' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RemoveTest_smithi111-76430-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1985010718' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RemoveTest_smithi111-76430-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1742049782' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-7"}]': finished 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi111-77719-3"}]': finished 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: osdmap e87: 8 total, 8 up, 8 in 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3735571060' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RemoveTest_smithi111-76430-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1985010718' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RemoveTest_smithi111-76430-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:16.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:17.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:17 smithi114 ceph-mon[33315]: pgmap v80: 680 pgs: 128 creating+peering, 552 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-16T09:01:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi111-76430-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:17 smithi114 ceph-mon[33315]: osdmap e88: 8 total, 8 up, 8 in 2024-09-16T09:01:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[29803]: pgmap v80: 680 pgs: 128 creating+peering, 552 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-16T09:01:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi111-76430-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[29803]: osdmap e88: 8 total, 8 up, 8 in 2024-09-16T09:01:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[40056]: pgmap v80: 680 pgs: 128 creating+peering, 552 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-16T09:01:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi111-76430-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:17 smithi111 ceph-mon[40056]: osdmap e88: 8 total, 8 up, 8 in 2024-09-16T09:01:18.247 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: Running main() from gmock_main.cc 2024-09-16T09:01:18.247 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-16T09:01:18.247 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [----------] Global test environment set-up. 2024-09-16T09:01:18.247 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (2653 ms) 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (2982 ms) 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (2273 ms) 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3014 ms) 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (2999 ms) 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (2999 ms) 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (6020 ms) 2024-09-16T09:01:18.248 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (22940 ms total) 2024-09-16T09:01:18.249 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: 2024-09-16T09:01:18.249 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [----------] Global test environment tear-down 2024-09-16T09:01:18.249 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (22940 ms total) 2024-09-16T09:01:18.249 INFO:tasks.workunit.client.0.smithi111.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-16T09:01:18.310 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-16T09:01:18.310 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-16T09:01:18.310 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-16T09:01:18.310 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2822 ms) 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3034 ms) 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2275 ms) 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3018 ms) 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-16T09:01:18.311 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2981 ms) 2024-09-16T09:01:18.312 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-16T09:01:18.312 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3000 ms) 2024-09-16T09:01:18.312 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-16T09:01:18.312 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (6075 ms) 2024-09-16T09:01:18.312 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (23205 ms total) 2024-09-16T09:01:18.312 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: 2024-09-16T09:01:18.312 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-16T09:01:18.312 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (23205 ms total) 2024-09-16T09:01:18.312 INFO:tasks.workunit.client.0.smithi111.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-16T09:01:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:18 smithi114 ceph-mon[33315]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:18 smithi114 ceph-mon[33315]: osdmap e89: 8 total, 8 up, 8 in 2024-09-16T09:01:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:18.925 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[29803]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[29803]: osdmap e89: 8 total, 8 up, 8 in 2024-09-16T09:01:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:18.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:18.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[40056]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:18.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[40056]: osdmap e89: 8 total, 8 up, 8 in 2024-09-16T09:01:18.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:18.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: pgmap v83: 520 pgs: 64 unknown, 456 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: osdmap e90: 8 total, 8 up, 8 in 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/772688078' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi111-76430-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1022688724' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi111-76430-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4"} : dispatch 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: pgmap v83: 520 pgs: 64 unknown, 456 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: osdmap e90: 8 total, 8 up, 8 in 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/772688078' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi111-76430-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1022688724' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi111-76430-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:19.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4"} : dispatch 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: pgmap v83: 520 pgs: 64 unknown, 456 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: osdmap e90: 8 total, 8 up, 8 in 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/772688078' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi111-76430-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1022688724' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi111-76430-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-5","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:19.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4"} : dispatch 2024-09-16T09:01:20.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi111-76430-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4"}]': finished 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: osdmap e91: 8 total, 8 up, 8 in 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/594923566' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6"} : dispatch 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi111-76430-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4"}]': finished 2024-09-16T09:01:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: osdmap e91: 8 total, 8 up, 8 in 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/594923566' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6"} : dispatch 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi111-76430-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi111-76450-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4"}]': finished 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: osdmap e91: 8 total, 8 up, 8 in 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/594923566' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:20.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6"} : dispatch 2024-09-16T09:01:20.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:20.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:21.815 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:01:21.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: pgmap v86: 616 pgs: 224 unknown, 392 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:21.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:21.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:21.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6"}]': finished 2024-09-16T09:01:21.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-9"}]': finished 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: osdmap e92: 8 total, 8 up, 8 in 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "test-rados-api-smithi111-77003-6", "pool2": "test-rados-api-smithi111-77003-6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-9", "mode": "writeback"} : dispatch 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-9", "mode": "writeback"} : dispatch 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:21.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: pgmap v86: 616 pgs: 224 unknown, 392 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6"}]': finished 2024-09-16T09:01:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-9"}]': finished 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: osdmap e92: 8 total, 8 up, 8 in 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "test-rados-api-smithi111-77003-6", "pool2": "test-rados-api-smithi111-77003-6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-9", "mode": "writeback"} : dispatch 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-9", "mode": "writeback"} : dispatch 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: pgmap v86: 616 pgs: 224 unknown, 392 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi111-77003-4", "tierpool": "test-rados-api-smithi111-77003-6"}]': finished 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-9"}]': finished 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: osdmap e92: 8 total, 8 up, 8 in 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "test-rados-api-smithi111-77003-6", "pool2": "test-rados-api-smithi111-77003-6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-9", "mode": "writeback"} : dispatch 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-9", "mode": "writeback"} : dispatch 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:21.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:21.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [----------] Global test environment set-up. 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5052 ms) 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [ OK ] LibRadosService.RegisterLate (22 ms) 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: cluster: 2024-09-16T09:01:22.378 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: id: f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: health: HEALTH_WARN 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 10 pool(s) do not have an application enabled 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: services: 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: mgr: y(active, since 89s), standbys: x 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: data: 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: pools: 24 pools, 636 pgs 2024-09-16T09:01:22.379 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: objects: 199 objects, 583 KiB 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: usage: 231 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: pgs: 23.899% pgs unknown 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 55.346% pgs not active 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 352 creating+peering 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 152 unknown 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 132 active+clean 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: io: 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: client: 852 B/s rd, 0 op/s rd, 0 op/s wr 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: cluster: 2024-09-16T09:01:22.380 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: id: f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: health: HEALTH_WARN 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 12 pool(s) do not have an application enabled 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: services: 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: mgr: y(active, since 91s), standbys: x 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: data: 2024-09-16T09:01:22.381 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: pools: 29 pools, 868 pgs 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: objects: 199 objects, 583 KiB 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: usage: 231 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: pgs: 58.986% pgs unknown 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 25.806% pgs not active 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 512 unknown 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 224 creating+peering 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 132 active+clean 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2179 ms) 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [ OK ] LibRadosService.Status (20040 ms) 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [----------] 4 tests from LibRadosService (27293 ms total) 2024-09-16T09:01:22.382 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: 2024-09-16T09:01:22.383 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [----------] Global test environment tear-down 2024-09-16T09:01:22.383 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27293 ms total) 2024-09-16T09:01:22.383 INFO:tasks.workunit.client.0.smithi111.stdout: api_service: [ PASSED ] 4 tests. 2024-09-16T09:01:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi111-77003-6", "pool2": "test-rados-api-smithi111-77003-6", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T09:01:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-9", "mode": "writeback"}]': finished 2024-09-16T09:01:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: osdmap e93: 8 total, 8 up, 8 in 2024-09-16T09:01:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi111-76789-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:22.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi111-76789-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:22.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4023661380' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattr_smithi111-76430-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattr_smithi111-76430-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1278118915' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi111-76450-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi111-76450-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:22.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi111-77003-6", "pool2": "test-rados-api-smithi111-77003-6", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-9", "mode": "writeback"}]': finished 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: osdmap e93: 8 total, 8 up, 8 in 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi111-76789-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi111-76789-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:22.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4023661380' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattr_smithi111-76430-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattr_smithi111-76430-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1278118915' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi111-76450-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi111-76450-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/752427736' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi111-77003-6", "pool2": "test-rados-api-smithi111-77003-6", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-9", "mode": "writeback"}]': finished 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: osdmap e93: 8 total, 8 up, 8 in 2024-09-16T09:01:22.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi111-76789-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:22.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi111-76789-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:22.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4023661380' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattr_smithi111-76430-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattr_smithi111-76430-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1278118915' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi111-76450-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi111-76450-6","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:22.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:22.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout:h_notify_pp: flushed 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3003 ms) 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: trying... 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605459696 notify_id 347892350984 notifier_gid 15273 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: timed out 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: flushing 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: flushed 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3004 ms) 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-16T09:01:23.264 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: List watches 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: notify2 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605447648 notify_id 360777252873 notifier_gid 15273 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: notify2 done 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: watch_check 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: unwatch2 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: flushing 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: done 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3019 ms) 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: List watches 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: notify2 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: handle_notify cookie 93839605447648 notify_id 373662154762 notifier_gid 15273 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: notify2 done 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: watch_check 2024-09-16T09:01:23.265 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: unwatch2 2024-09-16T09:01:23.266 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: flushing 2024-09-16T09:01:23.266 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: done 2024-09-16T09:01:23.266 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3021 ms) 2024-09-16T09:01:23.266 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16172 ms total) 2024-09-16T09:01:23.266 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: 2024-09-16T09:01:23.266 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-16T09:01:23.266 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28230 ms total) 2024-09-16T09:01:23.266 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-16T09:01:23.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[29803]: pgmap v89: 612 pgs: 32 creating+peering, 160 unknown, 420 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:23.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:23.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[29803]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:23.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi111-76430-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:23.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi111-76450-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:23.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:01:23.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:23.636 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:01:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:01:23.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:23 smithi114 ceph-mon[33315]: pgmap v89: 612 pgs: 32 creating+peering, 160 unknown, 420 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:23.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:23.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:23 smithi114 ceph-mon[33315]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi111-76430-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi111-76450-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:01:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:23 smithi114 ceph-mon[33315]: osdmap e94: 8 total, 8 up, 8 in 2024-09-16T09:01:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:23.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[29803]: osdmap e94: 8 total, 8 up, 8 in 2024-09-16T09:01:23.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:23.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[40056]: pgmap v89: 612 pgs: 32 creating+peering, 160 unknown, 420 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:23.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:23.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[40056]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:23.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi111-76430-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:23.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi111-76450-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:23.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:01:23.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:23.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[40056]: osdmap e94: 8 total, 8 up, 8 in 2024-09-16T09:01:23.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9"} : dispatch 2024-09-16T09:01:24.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:24.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:24 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:24 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi111-76789-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi111-76789-10"}]': finished 2024-09-16T09:01:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:24 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9"}]': finished 2024-09-16T09:01:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:24 smithi114 ceph-mon[33315]: osdmap e95: 8 total, 8 up, 8 in 2024-09-16T09:01:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi111-76789-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi111-76789-10"}]': finished 2024-09-16T09:01:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9"}]': finished 2024-09-16T09:01:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[40056]: osdmap e95: 8 total, 8 up, 8 in 2024-09-16T09:01:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:24.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi111-76789-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi111-76789-10"}]': finished 2024-09-16T09:01:24.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-9"}]': finished 2024-09-16T09:01:24.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:24 smithi111 ceph-mon[29803]: osdmap e95: 8 total, 8 up, 8 in 2024-09-16T09:01:25.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-16T09:01:25.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-16T09:01:25.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-16T09:01:25.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-16T09:01:25.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5076 ms) 2024-09-16T09:01:25.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-16T09:01:25.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (57 ms) 2024-09-16T09:01:25.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-16T09:01:25.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20071 ms) 2024-09-16T09:01:25.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-16T09:01:25.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: attempt 0 of 20 2024-09-16T09:01:25.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5159 ms) 2024-09-16T09:01:25.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30363 ms total) 2024-09-16T09:01:25.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: 2024-09-16T09:01:25.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-16T09:01:25.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30363 ms total) 2024-09-16T09:01:25.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-16T09:01:25.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:25 smithi114 ceph-mon[33315]: pgmap v92: 396 pgs: 32 creating+peering, 8 unknown, 356 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:25.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:25.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:25 smithi114 ceph-mon[33315]: osdmap e96: 8 total, 8 up, 8 in 2024-09-16T09:01:25.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2622127579' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP_smithi111-76450-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1160013444' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrIter_smithi111-76430-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP_smithi111-76450-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2720798841' entity='client.admin' cmd={"prefix": "service dump"} : dispatch 2024-09-16T09:01:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[29803]: pgmap v92: 396 pgs: 32 creating+peering, 8 unknown, 356 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[29803]: osdmap e96: 8 total, 8 up, 8 in 2024-09-16T09:01:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2622127579' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP_smithi111-76450-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1160013444' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrIter_smithi111-76430-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP_smithi111-76450-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2720798841' entity='client.admin' cmd={"prefix": "service dump"} : dispatch 2024-09-16T09:01:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[40056]: pgmap v92: 396 pgs: 32 creating+peering, 8 unknown, 356 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[40056]: osdmap e96: 8 total, 8 up, 8 in 2024-09-16T09:01:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2622127579' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP_smithi111-76450-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1160013444' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrIter_smithi111-76430-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP_smithi111-76450-7","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2720798841' entity='client.admin' cmd={"prefix": "service dump"} : dispatch 2024-09-16T09:01:26.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:26 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-16T09:01:26.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:26.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1160013444' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi111-76430-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi111-76450-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:26 smithi114 ceph-mon[33315]: osdmap e97: 8 total, 8 up, 8 in 2024-09-16T09:01:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-16T09:01:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1160013444' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi111-76430-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi111-76450-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[29803]: osdmap e97: 8 total, 8 up, 8 in 2024-09-16T09:01:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-16T09:01:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76760-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1160013444' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi111-76430-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi111-76450-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[40056]: osdmap e97: 8 total, 8 up, 8 in 2024-09-16T09:01:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: Running main() from gmock_main.cc 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [----------] Global test environment set-up. 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.Version (1664 ms) 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (1988 ms) 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-16T09:01:27.277 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.LongName (3020 ms) 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2274 ms) 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3040 ms) 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (2973 ms) 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.Exec (3010 ms) 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3036 ms) 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2961 ms) 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3001 ms) 2024-09-16T09:01:27.278 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-16T09:01:27.279 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2002 ms) 2024-09-16T09:01:27.279 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-16T09:01:27.279 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3029 ms) 2024-09-16T09:01:27.279 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [----------] 12 tests from NeoRadosMisc (31998 ms total) 2024-09-16T09:01:27.279 INFO:tasks.workunit.client.0.smithi111.stdout: misc: 2024-09-16T09:01:27.279 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [----------] Global test environment tear-down 2024-09-16T09:01:27.279 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [==========] 12 tests from 1 test suite ran. (31998 ms total) 2024-09-16T09:01:27.279 INFO:tasks.workunit.client.0.smithi111.stdout: misc: [ PASSED ] 12 tests. 2024-09-16T09:01:27.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:27 smithi114 ceph-mon[33315]: pgmap v95: 588 pgs: 64 creating+peering, 524 active+clean; 146 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 258 KiB/s wr, 12 op/s 2024-09-16T09:01:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:27 smithi114 ceph-mon[33315]: osdmap e98: 8 total, 8 up, 8 in 2024-09-16T09:01:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache", "force_nonempty":""} : dispatch 2024-09-16T09:01:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache", "force_nonempty":""} : dispatch 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[40056]: pgmap v95: 588 pgs: 64 creating+peering, 524 active+clean; 146 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 258 KiB/s wr, 12 op/s 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[40056]: osdmap e98: 8 total, 8 up, 8 in 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache", "force_nonempty":""} : dispatch 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache", "force_nonempty":""} : dispatch 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[29803]: pgmap v95: 588 pgs: 64 creating+peering, 524 active+clean; 146 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 258 KiB/s wr, 12 op/s 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[29803]: osdmap e98: 8 total, 8 up, 8 in 2024-09-16T09:01:27.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache", "force_nonempty":""} : dispatch 2024-09-16T09:01:27.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache", "force_nonempty":""} : dispatch 2024-09-16T09:01:28.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:28.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache", "force_nonempty":""}]': finished 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: osdmap e99: 8 total, 8 up, 8 in 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76760-10-cache", "mode":"readonly", "yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/703246951' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi111-76450-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi111-76450-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1370725804' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsComplete_smithi111-76430-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsComplete_smithi111-76430-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76760-10-cache", "mode":"readonly", "yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:28.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache", "force_nonempty":""}]': finished 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: osdmap e99: 8 total, 8 up, 8 in 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76760-10-cache", "mode":"readonly", "yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/703246951' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi111-76450-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi111-76450-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1370725804' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsComplete_smithi111-76430-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsComplete_smithi111-76430-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76760-10-cache", "mode":"readonly", "yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache", "force_nonempty":""}]': finished 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: osdmap e99: 8 total, 8 up, 8 in 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76760-10-cache", "mode":"readonly", "yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/703246951' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi111-76450-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi111-76450-8","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1370725804' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsComplete_smithi111-76430-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsComplete_smithi111-76430-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76760-10-cache", "mode":"readonly", "yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: pgmap v98: 588 pgs: 160 unknown, 64 creating+peering, 364 active+clean; 146 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi111-76450-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi111-76430-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76760-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: osdmap e100: 8 total, 8 up, 8 in 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache"} : dispatch 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache"} : dispatch 2024-09-16T09:01:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: pgmap v98: 588 pgs: 160 unknown, 64 creating+peering, 364 active+clean; 146 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-09-16T09:01:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi111-76450-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi111-76430-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76760-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: osdmap e100: 8 total, 8 up, 8 in 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache"} : dispatch 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache"} : dispatch 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: pgmap v98: 588 pgs: 160 unknown, 64 creating+peering, 364 active+clean; 146 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi111-76450-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi111-76430-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76760-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: osdmap e100: 8 total, 8 up, 8 in 2024-09-16T09:01:29.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/198205300' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache"} : dispatch 2024-09-16T09:01:29.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:29.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache"} : dispatch 2024-09-16T09:01:30.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:30.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.0"} : dispatch 2024-09-16T09:01:30.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-09-16T09:01:30.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.1"} : dispatch 2024-09-16T09:01:30.647 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-09-16T09:01:30.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.2"} : dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.3"} : dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.4"} : dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.5"} : dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.6"} : dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.7"} : dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.8"} : dispatch 2024-09-16T09:01:30.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-09-16T09:01:30.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-11"}]': finished 2024-09-16T09:01:30.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache"}]': finished 2024-09-16T09:01:30.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: osdmap e101: 8 total, 8 up, 8 in 2024-09-16T09:01:30.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.9"} : dispatch 2024-09-16T09:01:30.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-09-16T09:01:30.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-11", "mode": "writeback"} : dispatch 2024-09-16T09:01:30.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-11", "mode": "writeback"} : dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.0"} : dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.1"} : dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.2"} : dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.3"} : dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.4"} : dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-09-16T09:01:30.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.5"} : dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.6"} : dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.7"} : dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.8"} : dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-11"}]': finished 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache"}]': finished 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: osdmap e101: 8 total, 8 up, 8 in 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.9"} : dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-11", "mode": "writeback"} : dispatch 2024-09-16T09:01:30.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-11", "mode": "writeback"} : dispatch 2024-09-16T09:01:30.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:30.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.0"} : dispatch 2024-09-16T09:01:30.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.1"} : dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.2"} : dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.3"} : dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.4"} : dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.5"} : dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.6"} : dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.7"} : dispatch 2024-09-16T09:01:30.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-09-16T09:01:30.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.8"} : dispatch 2024-09-16T09:01:30.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-09-16T09:01:30.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-11"}]': finished 2024-09-16T09:01:30.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76760-10", "tierpool":"test-rados-api-smithi111-76760-10-cache"}]': finished 2024-09-16T09:01:30.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: osdmap e101: 8 total, 8 up, 8 in 2024-09-16T09:01:30.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "15.9"} : dispatch 2024-09-16T09:01:30.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-09-16T09:01:30.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-11", "mode": "writeback"} : dispatch 2024-09-16T09:01:30.931 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-11", "mode": "writeback"} : dispatch 2024-09-16T09:01:31.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:01:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: 15.3 deep-scrub starts 2024-09-16T09:01:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: 15.3 deep-scrub ok 2024-09-16T09:01:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: pgmap v101: 460 pgs: 32 unknown, 64 creating+peering, 364 active+clean; 146 MiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: 15.6 deep-scrub starts 2024-09-16T09:01:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: 15.6 deep-scrub ok 2024-09-16T09:01:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: 15.4 deep-scrub starts 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: 15.4 deep-scrub ok 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: 15.0 deep-scrub starts 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: 15.0 deep-scrub ok 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-11", "mode": "writeback"}]': finished 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2060964198' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: osdmap e102: 8 total, 8 up, 8 in 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/941391670' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafe_smithi111-76430-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafe_smithi111-76430-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: 15.3 deep-scrub starts 2024-09-16T09:01:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: 15.3 deep-scrub ok 2024-09-16T09:01:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: pgmap v101: 460 pgs: 32 unknown, 64 creating+peering, 364 active+clean; 146 MiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: 15.6 deep-scrub starts 2024-09-16T09:01:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: 15.6 deep-scrub ok 2024-09-16T09:01:31.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: 15.4 deep-scrub starts 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: 15.4 deep-scrub ok 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: 15.0 deep-scrub starts 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: 15.0 deep-scrub ok 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-11", "mode": "writeback"}]': finished 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2060964198' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: osdmap e102: 8 total, 8 up, 8 in 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/941391670' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafe_smithi111-76430-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafe_smithi111-76430-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: 15.3 deep-scrub starts 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: 15.3 deep-scrub ok 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: pgmap v101: 460 pgs: 32 unknown, 64 creating+peering, 364 active+clean; 146 MiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: 15.6 deep-scrub starts 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: 15.6 deep-scrub ok 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: 15.4 deep-scrub starts 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: 15.4 deep-scrub ok 2024-09-16T09:01:31.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: 15.0 deep-scrub starts 2024-09-16T09:01:31.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: 15.0 deep-scrub ok 2024-09-16T09:01:31.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:31.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-11", "mode": "writeback"}]': finished 2024-09-16T09:01:31.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2060964198' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: osdmap e102: 8 total, 8 up, 8 in 2024-09-16T09:01:31.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-9","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/941391670' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafe_smithi111-76430-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:31.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafe_smithi111-76430-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: Running main() from gmock_main.cc 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [----------] Global test environment set-up. 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2975 ms) 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3006 ms) 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2275 ms) 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3023 ms) 2024-09-16T09:01:32.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2990 ms) 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5015 ms) 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5007 ms) 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5001 ms) 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8000 ms) 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [----------] 9 tests from LibRadosPools (37292 ms total) 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [----------] Global test environment tear-down 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37292 ms total) 2024-09-16T09:01:32.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: 15.1 deep-scrub starts 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: 15.1 deep-scrub ok 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: 15.7 deep-scrub starts 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: 15.7 deep-scrub ok 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: 15.2 deep-scrub starts 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: 15.2 deep-scrub ok 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi111-76430-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: osdmap e103: 8 total, 8 up, 8 in 2024-09-16T09:01:32.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:32.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:32.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: 15.1 deep-scrub starts 2024-09-16T09:01:32.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: 15.1 deep-scrub ok 2024-09-16T09:01:32.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:32.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:32.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: 15.7 deep-scrub starts 2024-09-16T09:01:32.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: 15.7 deep-scrub ok 2024-09-16T09:01:32.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: 15.2 deep-scrub starts 2024-09-16T09:01:32.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: 15.2 deep-scrub ok 2024-09-16T09:01:32.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:32.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi111-76430-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:32.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: osdmap e103: 8 total, 8 up, 8 in 2024-09-16T09:01:32.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:32.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:32.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: 15.1 deep-scrub starts 2024-09-16T09:01:32.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: 15.1 deep-scrub ok 2024-09-16T09:01:32.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: 15.7 deep-scrub starts 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: 15.7 deep-scrub ok 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: 15.2 deep-scrub starts 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: 15.2 deep-scrub ok 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi111-76430-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: osdmap e103: 8 total, 8 up, 8 in 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:32.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:01:33.277 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-16T09:01:33.277 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] Global test environment set-up. 2024-09-16T09:01:33.277 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-16T09:01:33.277 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-16T09:01:33.277 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-16T09:01:33.278 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T09:01:33.278 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: 2024-09-16T09:01:33.278 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-16T09:01:33.278 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-16T09:01:33.278 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-16T09:01:33.278 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: 2024-09-16T09:00:55.007+0000 7f08a24b2780 -1 failed for service _ceph-mon._tcp 2024-09-16T09:01:33.278 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: 2024-09-16T09:00:55.007+0000 7f08a24b2780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-16T09:01:33.278 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (63 ms) 2024-09-16T09:01:33.279 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-16T09:01:33.279 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5020 ms) 2024-09-16T09:01:33.279 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5083 ms total) 2024-09-16T09:01:33.279 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: 2024-09-16T09:01:33.279 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-16T09:01:33.279 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-16T09:01:33.280 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: started 0x560a2e07e060 2024-09-16T09:01:33.280 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: started 0x560a2e1c7af0 2024-09-16T09:01:33.280 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: started 2 aios 2024-09-16T09:01:33.280 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: waiting 0x560a2e07e060 2024-09-16T09:01:33.280 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: waiting 0x560a2e1c7af0 2024-09-16T09:01:33.280 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: done. 2024-09-16T09:01:33.280 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5203 ms) 2024-09-16T09:01:33.281 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5203 ms total) 2024-09-16T09:01:33.281 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: 2024-09-16T09:01:33.281 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-16T09:01:33.281 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-16T09:01:33.281 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-16T09:01:33.281 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-16T09:01:33.282 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMisc.Exec (65 ms) 2024-09-16T09:01:33.282 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-16T09:01:33.282 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (25 ms) 2024-09-16T09:01:33.282 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-16T09:01:33.282 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (5 ms) 2024-09-16T09:01:33.283 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-16T09:01:33.283 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4891 ms) 2024-09-16T09:01:33.283 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-16T09:01:33.283 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-16T09:01:33.283 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-16T09:01:33.283 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-16T09:01:33.284 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-16T09:01:33.284 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (20578 ms) 2024-09-16T09:01:33.284 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] 8 tests from LibRadosMisc (25564 ms total) 2024-09-16T09:01:33.284 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: 2024-09-16T09:01:33.284 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [----------] Global test environment tear-down 2024-09-16T09:01:33.285 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (38305 ms total) 2024-09-16T09:01:33.285 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-16T09:01:33.649 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:01:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:01:33.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: 15.5 deep-scrub starts 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: 15.5 deep-scrub ok 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: pgmap v104: 524 pgs: 32 unknown, 128 creating+peering, 364 active+clean; 217 MiB data, 1003 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 9 op/s 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: 15.8 deep-scrub starts 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: 15.8 deep-scrub ok 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: osdmap e104: 8 total, 8 up, 8 in 2024-09-16T09:01:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:33.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: 15.5 deep-scrub starts 2024-09-16T09:01:33.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: 15.5 deep-scrub ok 2024-09-16T09:01:33.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: pgmap v104: 524 pgs: 32 unknown, 128 creating+peering, 364 active+clean; 217 MiB data, 1003 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 9 op/s 2024-09-16T09:01:33.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:33.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: 15.8 deep-scrub starts 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: 15.8 deep-scrub ok 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: osdmap e104: 8 total, 8 up, 8 in 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: 15.5 deep-scrub starts 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: 15.5 deep-scrub ok 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: pgmap v104: 524 pgs: 32 unknown, 128 creating+peering, 364 active+clean; 217 MiB data, 1003 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 9 op/s 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: 15.8 deep-scrub starts 2024-09-16T09:01:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: 15.8 deep-scrub ok 2024-09-16T09:01:33.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:33.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:01:33.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:33.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: osdmap e104: 8 total, 8 up, 8 in 2024-09-16T09:01:33.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11"} : dispatch 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: 15.9 deep-scrub starts 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: 15.9 deep-scrub ok 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11"}]': finished 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: osdmap e105: 8 total, 8 up, 8 in 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2756295600' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi111-76450-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi111-76450-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1031113850' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReturnValue_smithi111-76430-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReturnValue_smithi111-76430-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:34.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: 15.9 deep-scrub starts 2024-09-16T09:01:34.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: 15.9 deep-scrub ok 2024-09-16T09:01:34.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:34.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:34.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11"}]': finished 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: osdmap e105: 8 total, 8 up, 8 in 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2756295600' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi111-76450-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi111-76450-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1031113850' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReturnValue_smithi111-76430-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReturnValue_smithi111-76430-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: 15.9 deep-scrub starts 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: 15.9 deep-scrub ok 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:34.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:01:34.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-11"}]': finished 2024-09-16T09:01:34.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: osdmap e105: 8 total, 8 up, 8 in 2024-09-16T09:01:34.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2756295600' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi111-76450-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi111-76450-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1031113850' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReturnValue_smithi111-76430-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:34.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReturnValue_smithi111-76430-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:34.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:35.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:35 smithi114 ceph-mon[33315]: pgmap v107: 484 pgs: 160 unknown, 324 active+clean; 217 MiB data, 1003 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-09-16T09:01:35.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:35.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi111-76450-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:35.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi111-76430-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:35.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"}]': finished 2024-09-16T09:01:35.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:35.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:35 smithi114 ceph-mon[33315]: osdmap e106: 8 total, 8 up, 8 in 2024-09-16T09:01:35.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[29803]: pgmap v107: 484 pgs: 160 unknown, 324 active+clean; 217 MiB data, 1003 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-09-16T09:01:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi111-76450-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi111-76430-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"}]': finished 2024-09-16T09:01:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[29803]: osdmap e106: 8 total, 8 up, 8 in 2024-09-16T09:01:35.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:35.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[40056]: pgmap v107: 484 pgs: 160 unknown, 324 active+clean; 217 MiB data, 1003 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-09-16T09:01:35.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:35.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi111-76450-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:35.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi111-76430-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:35.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi111-76789-10"}]': finished 2024-09-16T09:01:35.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2179775893' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:35.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[40056]: osdmap e106: 8 total, 8 up, 8 in 2024-09-16T09:01:35.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"} : dispatch 2024-09-16T09:01:36.268 INFO:tasks.workunit.client.0.smithi111.stdout: io: Running main() from gmock_main.cc 2024-09-16T09:01:36.268 INFO:tasks.workunit.client.0.smithi111.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-16T09:01:36.268 INFO:tasks.workunit.client.0.smithi111.stdout: io: [----------] Global test environment set-up. 2024-09-16T09:01:36.268 INFO:tasks.workunit.client.0.smithi111.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-16T09:01:36.268 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-16T09:01:36.268 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.Limits (2755 ms) 2024-09-16T09:01:36.268 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-16T09:01:36.268 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3011 ms) 2024-09-16T09:01:36.268 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.ReadOp (2284 ms) 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.SparseRead (3014 ms) 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2999 ms) 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (2997 ms) 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3022 ms) 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2987 ms) 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3018 ms) 2024-09-16T09:01:36.269 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.Trunc (3046 ms) 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.Remove (3002 ms) 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2955 ms) 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.RmXattr (3001 ms) 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3015 ms) 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [----------] 14 tests from NeoRadosIo (41106 ms total) 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [----------] Global test environment tear-down 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [==========] 14 tests from 1 test suite ran. (41107 ms total) 2024-09-16T09:01:36.270 INFO:tasks.workunit.client.0.smithi111.stdout: io: [ PASSED ] 14 tests. 2024-09-16T09:01:36.277 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: Running main() from gmock_main.cc 2024-09-16T09:01:36.277 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-16T09:01:36.277 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [----------] Global test environment set-up. 2024-09-16T09:01:36.277 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2631 ms) 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3009 ms) 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (2279 ms) 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3023 ms) 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (2998 ms) 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (2994 ms) 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-16T09:01:36.278 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3032 ms) 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2976 ms) 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3008 ms) 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3009 ms) 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3030 ms) 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2963 ms) 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3013 ms) 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3019 ms) 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (40984 ms total) 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: 2024-09-16T09:01:36.279 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [----------] Global test environment tear-down 2024-09-16T09:01:36.280 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (40985 ms total) 2024-09-16T09:01:36.280 INFO:tasks.workunit.client.0.smithi111.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"}]': finished 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: osdmap e107: 8 total, 8 up, 8 in 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:36.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:36.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:36.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"}]': finished 2024-09-16T09:01:36.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: osdmap e107: 8 total, 8 up, 8 in 2024-09-16T09:01:36.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:36.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:36.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi111-76789-10"}]': finished 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: osdmap e107: 8 total, 8 up, 8 in 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:36.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:36.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:37.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: pgmap v110: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-09-16T09:01:37.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:37.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:37.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:37.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: osdmap e108: 8 total, 8 up, 8 in 2024-09-16T09:01:37.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:37.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:37.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1634568307' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "Flush_smithi111-76430-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:37.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3577930501' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsCompletePP_smithi111-76450-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:37.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsCompletePP_smithi111-76450-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:37.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:37 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: pgmap v110: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: osdmap e108: 8 total, 8 up, 8 in 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1634568307' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "Flush_smithi111-76430-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3577930501' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsCompletePP_smithi111-76450-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsCompletePP_smithi111-76450-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:37.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: pgmap v110: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: osdmap e108: 8 total, 8 up, 8 in 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1634568307' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "Flush_smithi111-76430-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3577930501' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsCompletePP_smithi111-76450-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsCompletePP_smithi111-76450-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:37.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:37 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:01:38.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:38.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1634568307' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi111-76430-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:38.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi111-76450-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:38.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:38 smithi114 ceph-mon[33315]: osdmap e109: 8 total, 8 up, 8 in 2024-09-16T09:01:38.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:38.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1634568307' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi111-76430-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:38.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi111-76450-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:38.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:38 smithi111 ceph-mon[40056]: osdmap e109: 8 total, 8 up, 8 in 2024-09-16T09:01:38.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:38.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1634568307' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi111-76430-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:38.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi111-76450-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:38.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:38 smithi111 ceph-mon[29803]: osdmap e109: 8 total, 8 up, 8 in 2024-09-16T09:01:39.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:39 smithi114 ceph-mon[33315]: pgmap v113: 420 pgs: 64 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 3.7 KiB/s wr, 3 op/s 2024-09-16T09:01:39.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:39 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:39.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:39 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:39.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:39 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"}]': finished 2024-09-16T09:01:39.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:39 smithi114 ceph-mon[33315]: osdmap e110: 8 total, 8 up, 8 in 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[40056]: pgmap v113: 420 pgs: 64 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 3.7 KiB/s wr, 3 op/s 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"}]': finished 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[40056]: osdmap e110: 8 total, 8 up, 8 in 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[29803]: pgmap v113: 420 pgs: 64 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 3.7 KiB/s wr, 3 op/s 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi111-76789-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"}]': finished 2024-09-16T09:01:39.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:39 smithi111 ceph-mon[29803]: osdmap e110: 8 total, 8 up, 8 in 2024-09-16T09:01:40.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:40.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:40 smithi114 ceph-mon[33315]: osdmap e111: 8 total, 8 up, 8 in 2024-09-16T09:01:40.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3660820415' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafePP_smithi111-76450-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafePP_smithi111-76450-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/866742131' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsync_smithi111-76430-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsync_smithi111-76430-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:40.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[29803]: osdmap e111: 8 total, 8 up, 8 in 2024-09-16T09:01:40.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3660820415' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafePP_smithi111-76450-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafePP_smithi111-76450-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/866742131' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsync_smithi111-76430-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsync_smithi111-76430-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:40.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[40056]: osdmap e111: 8 total, 8 up, 8 in 2024-09-16T09:01:40.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3660820415' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafePP_smithi111-76450-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "IsSafePP_smithi111-76450-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/866742131' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsync_smithi111-76430-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:40.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsync_smithi111-76430-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:41.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:01:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-mon[33315]: pgmap v116: 428 pgs: 104 unknown, 32 creating+peering, 292 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi111-76450-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi111-76430-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-mon[33315]: osdmap e112: 8 total, 8 up, 8 in 2024-09-16T09:01:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[40056]: pgmap v116: 428 pgs: 104 unknown, 32 creating+peering, 292 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi111-76450-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi111-76430-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[40056]: osdmap e112: 8 total, 8 up, 8 in 2024-09-16T09:01:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[29803]: pgmap v116: 428 pgs: 104 unknown, 32 creating+peering, 292 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi111-76450-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi111-76430-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[29803]: osdmap e112: 8 total, 8 up, 8 in 2024-09-16T09:01:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:01:43.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:43.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:43.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:01:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:42 smithi114 ceph-mon[33315]: osdmap e113: 8 total, 8 up, 8 in 2024-09-16T09:01:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[40056]: osdmap e113: 8 total, 8 up, 8 in 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:01:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[29803]: osdmap e113: 8 total, 8 up, 8 in 2024-09-16T09:01:43.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:01:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:01:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: pgmap v119: 364 pgs: 364 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 381 KiB/s wr, 122 op/s 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-13"}]': finished 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: osdmap e114: 8 total, 8 up, 8 in 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-13", "mode": "writeback"} : dispatch 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-13", "mode": "writeback"} : dispatch 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/691740394' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi111-76430-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3660996791' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi111-76450-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:44.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:01:44.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:01:44.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:43 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: pgmap v119: 364 pgs: 364 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 381 KiB/s wr, 122 op/s 2024-09-16T09:01:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-13"}]': finished 2024-09-16T09:01:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: osdmap e114: 8 total, 8 up, 8 in 2024-09-16T09:01:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-13", "mode": "writeback"} : dispatch 2024-09-16T09:01:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-13", "mode": "writeback"} : dispatch 2024-09-16T09:01:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/691740394' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi111-76430-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3660996791' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi111-76450-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:01:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:01:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:44.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: pgmap v119: 364 pgs: 364 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 381 KiB/s wr, 122 op/s 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-13"}]': finished 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: osdmap e114: 8 total, 8 up, 8 in 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-13", "mode": "writeback"} : dispatch 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-13", "mode": "writeback"} : dispatch 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/691740394' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi111-76430-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3660996791' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi111-76450-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:01:44.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:43 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:45.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:45.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:45.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:45.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:45.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-13", "mode": "writeback"}]': finished 2024-09-16T09:01:45.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/691740394' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi111-76430-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:45.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3660996791' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi111-76450-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:45.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"}]': finished 2024-09-16T09:01:45.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: osdmap e115: 8 total, 8 up, 8 in 2024-09-16T09:01:45.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:45.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:45.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:45.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:45.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:45.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:45.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:45.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:45.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-13", "mode": "writeback"}]': finished 2024-09-16T09:01:45.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/691740394' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi111-76430-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:45.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3660996791' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi111-76450-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:45.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"}]': finished 2024-09-16T09:01:45.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: osdmap e115: 8 total, 8 up, 8 in 2024-09-16T09:01:45.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:45.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:45.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:45.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:45.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:45.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-13", "mode": "writeback"}]': finished 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/691740394' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi111-76430-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3660996791' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi111-76450-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"}]': finished 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: osdmap e115: 8 total, 8 up, 8 in 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:45.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: pgmap v122: 428 pgs: 64 unknown, 364 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 385 KiB/s wr, 127 op/s 2024-09-16T09:01:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.0"} : dispatch 2024-09-16T09:01:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.0"}]: dispatch 2024-09-16T09:01:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.1"} : dispatch 2024-09-16T09:01:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.1"}]: dispatch 2024-09-16T09:01:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.2"} : dispatch 2024-09-16T09:01:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.2"}]: dispatch 2024-09-16T09:01:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.3"} : dispatch 2024-09-16T09:01:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.3"}]: dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.4"} : dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.4"}]: dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.5"} : dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.5"}]: dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.6"} : dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.6"}]: dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.7"} : dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.7"}]: dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.8"} : dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.8"}]: dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.9"} : dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.9"}]: dispatch 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.9 scrub starts 2024-09-16T09:01:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.2 scrub starts 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.2 scrub ok 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.9 scrub ok 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"}]': finished 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: osdmap e116: 8 total, 8 up, 8 in 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.0 scrub starts 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.0 scrub ok 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.4 scrub starts 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.4 scrub ok 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.8 scrub starts 2024-09-16T09:01:46.166 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:45 smithi114 ceph-mon[33315]: 172.8 scrub ok 2024-09-16T09:01:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: pgmap v122: 428 pgs: 64 unknown, 364 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 385 KiB/s wr, 127 op/s 2024-09-16T09:01:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.0"} : dispatch 2024-09-16T09:01:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.0"}]: dispatch 2024-09-16T09:01:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.1"} : dispatch 2024-09-16T09:01:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.1"}]: dispatch 2024-09-16T09:01:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.2"} : dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.2"}]: dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.3"} : dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.3"}]: dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.4"} : dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.4"}]: dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.5"} : dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.5"}]: dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.6"} : dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.6"}]: dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.7"} : dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.7"}]: dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.8"} : dispatch 2024-09-16T09:01:46.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.8"}]: dispatch 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.9"} : dispatch 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.9"}]: dispatch 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.9 scrub starts 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.2 scrub starts 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.2 scrub ok 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.9 scrub ok 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"}]': finished 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: osdmap e116: 8 total, 8 up, 8 in 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.0 scrub starts 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.0 scrub ok 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.4 scrub starts 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.4 scrub ok 2024-09-16T09:01:46.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.8 scrub starts 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[29803]: 172.8 scrub ok 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: pgmap v122: 428 pgs: 64 unknown, 364 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 385 KiB/s wr, 127 op/s 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.0"} : dispatch 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.0"}]: dispatch 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.1"} : dispatch 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.1"}]: dispatch 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.2"} : dispatch 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.2"}]: dispatch 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.3"} : dispatch 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.3"}]: dispatch 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.4"} : dispatch 2024-09-16T09:01:46.179 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.4"}]: dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.5"} : dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.5"}]: dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.6"} : dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.6"}]: dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.7"} : dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.7"}]: dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.8"} : dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.8"}]: dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "172.9"} : dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "172.9"}]: dispatch 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.9 scrub starts 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.2 scrub starts 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.2 scrub ok 2024-09-16T09:01:46.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.9 scrub ok 2024-09-16T09:01:46.181 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"}]': finished 2024-09-16T09:01:46.181 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: osdmap e116: 8 total, 8 up, 8 in 2024-09-16T09:01:46.181 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:46.181 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.0 scrub starts 2024-09-16T09:01:46.181 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.0 scrub ok 2024-09-16T09:01:46.181 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.4 scrub starts 2024-09-16T09:01:46.181 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.4 scrub ok 2024-09-16T09:01:46.181 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.8 scrub starts 2024-09-16T09:01:46.181 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:45 smithi111 ceph-mon[40056]: 172.8 scrub ok 2024-09-16T09:01:47.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: 172.7 scrub starts 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: 172.7 scrub ok 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: 172.1 scrub starts 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: 172.3 scrub starts 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: 172.3 scrub ok 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: osdmap e117: 8 total, 8 up, 8 in 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2562709754' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushPP_smithi111-76450-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2692861600' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi111-76430-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushPP_smithi111-76450-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:46 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi111-76430-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: 172.7 scrub starts 2024-09-16T09:01:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: 172.7 scrub ok 2024-09-16T09:01:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: 172.1 scrub starts 2024-09-16T09:01:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: 172.3 scrub starts 2024-09-16T09:01:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: 172.3 scrub ok 2024-09-16T09:01:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: osdmap e117: 8 total, 8 up, 8 in 2024-09-16T09:01:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2562709754' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushPP_smithi111-76450-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2692861600' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi111-76430-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushPP_smithi111-76450-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi111-76430-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: 172.7 scrub starts 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: 172.7 scrub ok 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: 172.1 scrub starts 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:01:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: 172.3 scrub starts 2024-09-16T09:01:47.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: 172.3 scrub ok 2024-09-16T09:01:47.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: osdmap e117: 8 total, 8 up, 8 in 2024-09-16T09:01:47.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:47.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2562709754' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushPP_smithi111-76450-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2692861600' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi111-76430-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushPP_smithi111-76450-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:47.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:46 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi111-76430-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:48.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:47 smithi114 ceph-mon[33315]: 172.1 scrub ok 2024-09-16T09:01:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:47 smithi114 ceph-mon[33315]: pgmap v125: 428 pgs: 96 creating+peering, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-16T09:01:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:47 smithi114 ceph-mon[33315]: 172.5 scrub starts 2024-09-16T09:01:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:47 smithi114 ceph-mon[33315]: 172.5 scrub ok 2024-09-16T09:01:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:47 smithi114 ceph-mon[33315]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:47 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi111-76450-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:47 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi111-76430-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:47 smithi114 ceph-mon[33315]: osdmap e118: 8 total, 8 up, 8 in 2024-09-16T09:01:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[29803]: 172.1 scrub ok 2024-09-16T09:01:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[29803]: pgmap v125: 428 pgs: 96 creating+peering, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-16T09:01:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[29803]: 172.5 scrub starts 2024-09-16T09:01:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[29803]: 172.5 scrub ok 2024-09-16T09:01:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[29803]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi111-76450-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi111-76430-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[29803]: osdmap e118: 8 total, 8 up, 8 in 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[40056]: 172.1 scrub ok 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[40056]: pgmap v125: 428 pgs: 96 creating+peering, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[40056]: 172.5 scrub starts 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[40056]: 172.5 scrub ok 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[40056]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi111-76450-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi111-76430-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:47 smithi111 ceph-mon[40056]: osdmap e118: 8 total, 8 up, 8 in 2024-09-16T09:01:49.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:48 smithi114 ceph-mon[33315]: 172.6 scrub starts 2024-09-16T09:01:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:48 smithi114 ceph-mon[33315]: 172.6 scrub ok 2024-09-16T09:01:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:48 smithi114 ceph-mon[33315]: osdmap e119: 8 total, 8 up, 8 in 2024-09-16T09:01:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:48 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[29803]: 172.6 scrub starts 2024-09-16T09:01:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[29803]: 172.6 scrub ok 2024-09-16T09:01:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[29803]: osdmap e119: 8 total, 8 up, 8 in 2024-09-16T09:01:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[40056]: 172.6 scrub starts 2024-09-16T09:01:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[40056]: 172.6 scrub ok 2024-09-16T09:01:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[40056]: osdmap e119: 8 total, 8 up, 8 in 2024-09-16T09:01:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:50.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-16T09:01:50.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-16T09:01:50.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-16T09:01:50.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-16T09:01:50.230 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2007 ms) 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2007 ms) 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2021 ms) 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1261 ms) 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (7296 ms total) 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4042 ms) 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4059 ms) 2024-09-16T09:01:50.231 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4963 ms) 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13064 ms total) 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3043 ms) 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1982 ms) 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1984 ms) 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2004 ms) 2024-09-16T09:01:50.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9013 ms total) 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4024 ms) 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4049 ms) 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8073 ms total) 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (55273 ms total) 2024-09-16T09:01:50.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-16T09:01:50.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: pgmap v128: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-09-16T09:01:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"}]': finished 2024-09-16T09:01:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: osdmap e120: 8 total, 8 up, 8 in 2024-09-16T09:01:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3203403533' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi111-76450-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1237766753' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStat_smithi111-76430-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi111-76450-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStat_smithi111-76430-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: pgmap v128: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-09-16T09:01:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"}]': finished 2024-09-16T09:01:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: osdmap e120: 8 total, 8 up, 8 in 2024-09-16T09:01:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3203403533' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi111-76450-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1237766753' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStat_smithi111-76430-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi111-76450-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStat_smithi111-76430-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: pgmap v128: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"}]': finished 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: osdmap e120: 8 total, 8 up, 8 in 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1011353480' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3203403533' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi111-76450-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1237766753' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStat_smithi111-76430-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"} : dispatch 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi111-76450-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStat_smithi111-76430-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:50.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:51.564 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"}]': finished 2024-09-16T09:01:51.564 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi111-76450-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:51.564 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi111-76430-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:51 smithi114 ceph-mon[33315]: osdmap e121: 8 total, 8 up, 8 in 2024-09-16T09:01:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:51 smithi114 ceph-mon[33315]: osdmap e122: 8 total, 8 up, 8 in 2024-09-16T09:01:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1842008521' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76863-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"}]': finished 2024-09-16T09:01:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi111-76450-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi111-76430-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[29803]: osdmap e121: 8 total, 8 up, 8 in 2024-09-16T09:01:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[29803]: osdmap e122: 8 total, 8 up, 8 in 2024-09-16T09:01:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1842008521' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76863-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi111-76789-15"}]': finished 2024-09-16T09:01:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi111-76450-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi111-76430-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[40056]: osdmap e121: 8 total, 8 up, 8 in 2024-09-16T09:01:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[40056]: osdmap e122: 8 total, 8 up, 8 in 2024-09-16T09:01:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1842008521' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76863-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:51.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:01:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:01:52.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:52 smithi114 ceph-mon[33315]: pgmap v131: 388 pgs: 96 unknown, 292 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:52.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1842008521' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76863-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:52 smithi114 ceph-mon[33315]: osdmap e123: 8 total, 8 up, 8 in 2024-09-16T09:01:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3559795319' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi111-76450-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3571102123' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76430-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76430-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[29803]: pgmap v131: 388 pgs: 96 unknown, 292 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1842008521' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76863-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[29803]: osdmap e123: 8 total, 8 up, 8 in 2024-09-16T09:01:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3559795319' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi111-76450-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3571102123' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76430-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76430-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[40056]: pgmap v131: 388 pgs: 96 unknown, 292 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1842008521' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76863-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[40056]: osdmap e123: 8 total, 8 up, 8 in 2024-09-16T09:01:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3559795319' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi111-76450-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3571102123' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76430-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76430-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:53.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:53 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3559795319' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi111-76450-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76430-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:53 smithi114 ceph-mon[33315]: osdmap e124: 8 total, 8 up, 8 in 2024-09-16T09:01:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:01:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:01:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3559795319' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi111-76450-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76430-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[29803]: osdmap e124: 8 total, 8 up, 8 in 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3559795319' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi111-76450-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76430-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:53 smithi111 ceph-mon[40056]: osdmap e124: 8 total, 8 up, 8 in 2024-09-16T09:01:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:54 smithi114 ceph-mon[33315]: pgmap v134: 420 pgs: 64 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:54 smithi114 ceph-mon[33315]: osdmap e125: 8 total, 8 up, 8 in 2024-09-16T09:01:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:54 smithi111 ceph-mon[29803]: pgmap v134: 420 pgs: 64 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:54 smithi111 ceph-mon[29803]: osdmap e125: 8 total, 8 up, 8 in 2024-09-16T09:01:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:54.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:54 smithi111 ceph-mon[40056]: pgmap v134: 420 pgs: 64 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:01:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:54 smithi111 ceph-mon[40056]: osdmap e125: 8 total, 8 up, 8 in 2024-09-16T09:01:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: pgmap v137: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: osdmap e126: 8 total, 8 up, 8 in 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2728934772' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi111-76430-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2005622360' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:56.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: pgmap v137: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: osdmap e126: 8 total, 8 up, 8 in 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2728934772' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi111-76430-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2005622360' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: pgmap v137: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: osdmap e126: 8 total, 8 up, 8 in 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2728934772' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi111-76430-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2005622360' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:56.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:01:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2728934772' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi111-76430-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:57 smithi114 ceph-mon[33315]: osdmap e127: 8 total, 8 up, 8 in 2024-09-16T09:01:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi111-76863-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi111-76863-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:57 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-16T09:01:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2728934772' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi111-76430-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[29803]: osdmap e127: 8 total, 8 up, 8 in 2024-09-16T09:01:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi111-76863-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi111-76863-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-16T09:01:57.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2728934772' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi111-76430-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:01:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[40056]: osdmap e127: 8 total, 8 up, 8 in 2024-09-16T09:01:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi111-76863-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi111-76863-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:01:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:57 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-16T09:01:58.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: pgmap v140: 388 pgs: 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:01:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-16T09:01:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: osdmap e128: 8 total, 8 up, 8 in 2024-09-16T09:01:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi111-76863-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"}]': finished 2024-09-16T09:01:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: osdmap e129: 8 total, 8 up, 8 in 2024-09-16T09:01:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1265095357' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi111-76450-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2008251456' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi111-76430-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi111-76450-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi111-76430-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: pgmap v140: 388 pgs: 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: osdmap e128: 8 total, 8 up, 8 in 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi111-76863-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"}]': finished 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: osdmap e129: 8 total, 8 up, 8 in 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1265095357' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi111-76450-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2008251456' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi111-76430-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi111-76450-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi111-76430-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: pgmap v140: 388 pgs: 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: osdmap e128: 8 total, 8 up, 8 in 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi111-76863-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"}]': finished 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: osdmap e129: 8 total, 8 up, 8 in 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1265095357' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi111-76450-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2008251456' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi111-76430-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi111-76450-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi111-76430-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:01:59.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi111-76450-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi111-76430-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"}]': finished 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: osdmap e130: 8 total, 8 up, 8 in 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:01:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi111-76450-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi111-76430-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"}]': finished 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: osdmap e130: 8 total, 8 up, 8 in 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"} : dispatch 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi111-76450-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi111-76430-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pg_num","val":"11"}]': finished 2024-09-16T09:01:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: osdmap e130: 8 total, 8 up, 8 in 2024-09-16T09:01:59.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:59.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:01:59.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:01:59.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:01:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"} : dispatch 2024-09-16T09:02:00.577 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[29803]: pgmap v143: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:00.577 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[29803]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:00.578 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"}]': finished 2024-09-16T09:02:00.578 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[29803]: osdmap e131: 8 total, 8 up, 8 in 2024-09-16T09:02:00.578 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:00.578 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[40056]: pgmap v143: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:00.578 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[40056]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:00.578 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"}]': finished 2024-09-16T09:02:00.578 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[40056]: osdmap e131: 8 total, 8 up, 8 in 2024-09-16T09:02:00.578 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:00.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:00 smithi114 ceph-mon[33315]: pgmap v143: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:00.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:00 smithi114 ceph-mon[33315]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi111-76517-1","var":"pgp_num","val":"11"}]': finished 2024-09-16T09:02:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:00 smithi114 ceph-mon[33315]: osdmap e131: 8 total, 8 up, 8 in 2024-09-16T09:02:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:01.575 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: osdmap e132: 8 total, 8 up, 8 in 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2236035450' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "StatRemove_smithi111-76430-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4146224784' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi111-76450-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi111-76450-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2"} : dispatch 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.576 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:01.577 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-16T09:02:01.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:01.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:01.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: osdmap e132: 8 total, 8 up, 8 in 2024-09-16T09:02:01.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2236035450' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "StatRemove_smithi111-76430-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4146224784' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi111-76450-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi111-76450-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2"} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app1","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: osdmap e132: 8 total, 8 up, 8 in 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2236035450' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "StatRemove_smithi111-76430-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.599 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4146224784' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi111-76450-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi111-76450-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2"} : dispatch 2024-09-16T09:02:01.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:01.600 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:01.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:02:01 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:02:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[29803]: pgmap v146: 332 pgs: 8 unknown, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2236035450' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi111-76430-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi111-76450-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[29803]: osdmap e133: 8 total, 8 up, 8 in 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"dne","key":"key1","value":"value1"} : dispatch 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[40056]: pgmap v146: 332 pgs: 8 unknown, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2236035450' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi111-76430-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi111-76450-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[40056]: osdmap e133: 8 total, 8 up, 8 in 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"dne","key":"key1","value":"value1"} : dispatch 2024-09-16T09:02:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:02:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:02.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:02:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:02 smithi114 ceph-mon[33315]: pgmap v146: 332 pgs: 8 unknown, 324 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2236035450' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi111-76430-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi111-76450-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi111-76617-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:02 smithi114 ceph-mon[33315]: osdmap e133: 8 total, 8 up, 8 in 2024-09-16T09:02:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"dne","key":"key1","value":"value1"} : dispatch 2024-09-16T09:02:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:02:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1","value":"value1"} : dispatch 2024-09-16T09:02:03.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-09-16T09:02:03.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-16T09:02:03.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-16T09:02:03.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-16T09:02:03.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: seed 76863 2024-09-16T09:02:03.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-16T09:02:03.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (1984 ms) 2024-09-16T09:02:03.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-16T09:02:03.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (1999 ms) 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2017 ms) 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1280 ms) 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3991 ms) 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11271 ms total) 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4108 ms) 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3907 ms) 2024-09-16T09:02:03.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6011 ms) 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4006 ms) 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2090 ms) 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4917-g8293d73f/rpm/el9/BUILD/ceph-19.3.0-4917-g8293d73f/src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi111-76863-7 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17910 ms) 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38032 ms total) 2024-09-16T09:02:03.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5026 ms) 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5035 ms total) 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2996 ms) 2024-09-16T09:02:03.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-16T09:02:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:03 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:03 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-16T09:02:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-16T09:02:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:03 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-16T09:02:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:03 smithi114 ceph-mon[33315]: osdmap e134: 8 total, 8 up, 8 in 2024-09-16T09:02:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:02:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:02:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:02:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:02:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-16T09:02:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-16T09:02:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-16T09:02:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[40056]: osdmap e134: 8 total, 8 up, 8 in 2024-09-16T09:02:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[29803]: osdmap e134: 8 total, 8 up, 8 in 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key2","value":"value2"} : dispatch 2024-09-16T09:02:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: pgmap v149: 396 pgs: 64 creating+peering, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-09-16T09:02:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-16T09:02:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: osdmap e135: 8 total, 8 up, 8 in 2024-09-16T09:02:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2203259707' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi111-76450-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4122569755' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClass_smithi111-76430-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi111-76450-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClass_smithi111-76430-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:02:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: pgmap v149: 396 pgs: 64 creating+peering, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: osdmap e135: 8 total, 8 up, 8 in 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2203259707' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi111-76450-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4122569755' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClass_smithi111-76430-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi111-76450-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClass_smithi111-76430-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:02:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: pgmap v149: 396 pgs: 64 creating+peering, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: osdmap e135: 8 total, 8 up, 8 in 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2203259707' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi111-76450-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4122569755' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClass_smithi111-76430-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi111-76450-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClass_smithi111-76430-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3560640447' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:02:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1"} : dispatch 2024-09-16T09:02:06.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosS api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-16T09:02:06.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-16T09:02:06.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-16T09:02:06.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-16T09:02:06.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-16T09:02:06.232 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: seed 76617 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (382 ms) 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (16 ms) 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (8 ms) 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (13 ms) 2024-09-16T09:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (4 ms) 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5 ms) 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (9 ms) 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-16T09:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (8 ms) 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1783 ms) 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (353 ms) 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: done waiting, doing copies 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: done waiting 2024-09-16T09:02:06.235 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60726 ms) 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (19 ms) 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (12 ms) 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4976 ms) 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-16T09:02:06.236 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-16T09:02:06.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:06 smithi114 ceph-mon[33315]: pgmap v152: 396 pgs: 64 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-09-16T09:02:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi111-76450-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi111-76430-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1"}]': finished 2024-09-16T09:02:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:06 smithi114 ceph-mon[33315]: osdmap e136: 8 total, 8 up, 8 in 2024-09-16T09:02:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:06 smithi114 ceph-mon[33315]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[29803]: pgmap v152: 396 pgs: 64 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-09-16T09:02:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi111-76450-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi111-76430-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1"}]': finished 2024-09-16T09:02:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[29803]: osdmap e136: 8 total, 8 up, 8 in 2024-09-16T09:02:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[29803]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:06.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[40056]: pgmap v152: 396 pgs: 64 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.5 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-09-16T09:02:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi111-76450-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi111-76430-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi111-76617-1","app":"app1","key":"key1"}]': finished 2024-09-16T09:02:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[40056]: osdmap e136: 8 total, 8 up, 8 in 2024-09-16T09:02:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:06 smithi111 ceph-mon[40056]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: osdmap e137: 8 total, 8 up, 8 in 2024-09-16T09:02:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76617-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76617-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "30"} : dispatch 2024-09-16T09:02:07.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: osdmap e137: 8 total, 8 up, 8 in 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76617-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76617-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "30"} : dispatch 2024-09-16T09:02:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: osdmap e137: 8 total, 8 up, 8 in 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76617-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76617-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "30"} : dispatch 2024-09-16T09:02:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: pgmap v155: 299 pgs: 299 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76617-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: osdmap e138: 8 total, 8 up, 8 in 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76617-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76617-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/461579162' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWrite_smithi111-76430-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3007404024' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi111-76450-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi111-76450-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWrite_smithi111-76430-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi111-76450-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi111-76430-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: osdmap e139: 8 total, 8 up, 8 in 2024-09-16T09:02:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: pgmap v155: 299 pgs: 299 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76617-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-09-16T09:02:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: osdmap e138: 8 total, 8 up, 8 in 2024-09-16T09:02:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76617-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76617-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/461579162' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWrite_smithi111-76430-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3007404024' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi111-76450-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi111-76450-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWrite_smithi111-76430-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi111-76450-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi111-76430-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: osdmap e139: 8 total, 8 up, 8 in 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: pgmap v155: 299 pgs: 299 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76617-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi111-76517-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: osdmap e138: 8 total, 8 up, 8 in 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76617-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76617-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/461579162' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWrite_smithi111-76430-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3007404024' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi111-76450-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi111-76450-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWrite_smithi111-76430-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi111-76450-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi111-76430-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: osdmap e139: 8 total, 8 up, 8 in 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/452275302' entity='client.admin' cmd={"prefix":"status","format":"json"} : dispatch 2024-09-16T09:02:09.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP snapshots: Running main() from gmock_main.cc 2024-09-16T09:02:09.227 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [----------] Global test environment set-up. 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4634 ms) 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (4279 ms) 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4017 ms) 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5008 ms) 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7004 ms) 2024-09-16T09:02:09.228 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (24942 ms total) 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5024 ms) 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6009 ms) 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7967 ms) 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6012 ms) 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3992 ms) 2024-09-16T09:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-16T09:02:09.230 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi111-77803-11. 2024-09-16T09:02:09.230 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: Waiting for snaps to purge. 2024-09-16T09:02:09.230 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19988 ms) 2024-09-16T09:02:09.230 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48992 ms total) 2024-09-16T09:02:09.230 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: 2024-09-16T09:02:09.230 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [----------] Global test environment tear-down 2024-09-16T09:02:09.230 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (73935 ms total) 2024-09-16T09:02:09.230 INFO:tasks.workunit.client.0.smithi111.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-16T09:02:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: pgmap v158: 355 pgs: 64 unknown, 291 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76617-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76617-24"}]': finished 2024-09-16T09:02:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"}]': finished 2024-09-16T09:02:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: osdmap e140: 8 total, 8 up, 8 in 2024-09-16T09:02:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi111-76517-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:10.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi111-76517-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: pgmap v158: 355 pgs: 64 unknown, 291 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76617-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76617-24"}]': finished 2024-09-16T09:02:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"}]': finished 2024-09-16T09:02:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: osdmap e140: 8 total, 8 up, 8 in 2024-09-16T09:02:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi111-76517-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi111-76517-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: pgmap v158: 355 pgs: 64 unknown, 291 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76617-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76617-24"}]': finished 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi111-76863-16"}]': finished 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: osdmap e140: 8 total, 8 up, 8 in 2024-09-16T09:02:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3124035360' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:10.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"} : dispatch 2024-09-16T09:02:10.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:10.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi111-76517-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:10.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi111-76517-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:11.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"}]': finished 2024-09-16T09:02:11.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi111-76517-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:11.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: osdmap e141: 8 total, 8 up, 8 in 2024-09-16T09:02:11.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosListEC_smithi111-76517-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:11.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosListEC_smithi111-76517-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3472365806' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76450-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76450-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3590792788' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlock_smithi111-76430-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlock_smithi111-76430-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:11.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76450-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"}]': finished 2024-09-16T09:02:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi111-76517-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: osdmap e141: 8 total, 8 up, 8 in 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosListEC_smithi111-76517-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosListEC_smithi111-76517-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3472365806' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76450-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76450-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3590792788' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlock_smithi111-76430-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlock_smithi111-76430-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76450-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi111-76863-16"}]': finished 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi111-76517-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: osdmap e141: 8 total, 8 up, 8 in 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosListEC_smithi111-76517-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosListEC_smithi111-76517-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3472365806' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76450-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76450-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3590792788' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlock_smithi111-76430-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlock_smithi111-76430-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:11.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:11.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:11.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:11.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi111-76450-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:11.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:02:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:02:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: pgmap v161: 332 pgs: 104 unknown, 228 active+clean; 590 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi111-76430-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: osdmap e142: 8 total, 8 up, 8 in 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi111-76517-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi111-76517-2"}]': finished 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:12 smithi114 ceph-mon[33315]: osdmap e143: 8 total, 8 up, 8 in 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: pgmap v161: 332 pgs: 104 unknown, 228 active+clean; 590 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi111-76430-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: osdmap e142: 8 total, 8 up, 8 in 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi111-76517-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi111-76517-2"}]': finished 2024-09-16T09:02:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[29803]: osdmap e143: 8 total, 8 up, 8 in 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: pgmap v161: 332 pgs: 104 unknown, 228 active+clean; 590 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi111-76430-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: osdmap e142: 8 total, 8 up, 8 in 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi111-76517-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi111-76517-2"}]': finished 2024-09-16T09:02:12.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76617-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:12.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:12 smithi111 ceph-mon[40056]: osdmap e143: 8 total, 8 up, 8 in 2024-09-16T09:02:13.295 INFO:tasks.workunit.client.0.smithi111.stdout:ry=13 expected=13 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:de5d7c5f:::12:head expected=13:de5d7c5f:::12:head 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=12 expected=12 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:d83876eb:::4:head expected=13:d83876eb:::4:head 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=4 expected=4 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:cfc208b3:::3:head expected=13:cfc208b3:::3:head 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-09-16T09:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=3 expected=3 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:bd63b0f1:::8:head expected=13:bd63b0f1:::8:head 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=8 expected=8 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:b29083e3:::5:head expected=13:b29083e3:::5:head 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=5 expected=5 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:89d3ae78:::11:head expected=13:89d3ae78:::11:head 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=11 expected=11 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:863748b0:::15:head 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:863748b0:::15:head expected=13:863748b0:::15:head 2024-09-16T09:02:13.297 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=15 expected=15 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : cursor()=13:6cac518f:::0:head expected=13:6cac518f:::0:head 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: : entry=0 expected=0 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (217 ms) 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (60533 ms) 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: split 0/5 -> MIN 13:33333333::::head 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: split 1/5 -> 13:33333333::::head 13:66666666::::head 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: split 2/5 -> 13:66666666::::head 13:99999999::::head 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: split 3/5 -> 13:99999999::::head 13:cccccccc::::head 2024-09-16T09:02:13.298 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: split 4/5 -> 13:cccccccc::::head MAX 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (10029 ms) 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [----------] 7 tests from LibRadosList (71386 ms total) 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1024 ms) 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo1 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo2 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo3 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo4 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo5 2024-09-16T09:02:13.299 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo1 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: myset foo6,foo7 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo7 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: foo6 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns1:foo4 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns1:foo5 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns2:foo7 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns2:foo6 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: ns1:foo1 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: :foo1 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: :foo2 2024-09-16T09:02:13.300 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: :foo3 2024-09-16T09:02:13.301 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (33 ms) 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi111-76430-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi111-76430-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"}]': finished 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi111-76430-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: osdmap e144: 8 total, 8 up, 8 in 2024-09-16T09:02:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWrite_smithi111-76430-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWrite_smithi111-76430-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3113454050' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime2_smithi111-76450-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime2_smithi111-76450-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:13.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:02:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:02:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi111-76430-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi111-76430-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"}]': finished 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi111-76430-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: osdmap e144: 8 total, 8 up, 8 in 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWrite_smithi111-76430-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWrite_smithi111-76430-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3113454050' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime2_smithi111-76450-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime2_smithi111-76450-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi111-76430-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi111-76430-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"}]': finished 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi111-76430-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: osdmap e144: 8 total, 8 up, 8 in 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWrite_smithi111-76430-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWrite_smithi111-76430-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3113454050' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime2_smithi111-76450-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:13.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OperateMtime2_smithi111-76450-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:14 smithi114 ceph-mon[33315]: pgmap v164: 276 pgs: 8 unknown, 268 active+clean; 590 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi111-76450-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:14 smithi114 ceph-mon[33315]: osdmap e145: 8 total, 8 up, 8 in 2024-09-16T09:02:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[40056]: pgmap v164: 276 pgs: 8 unknown, 268 active+clean; 590 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi111-76450-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[40056]: osdmap e145: 8 total, 8 up, 8 in 2024-09-16T09:02:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[29803]: pgmap v164: 276 pgs: 8 unknown, 268 active+clean; 590 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi111-76450-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[29803]: osdmap e145: 8 total, 8 up, 8 in 2024-09-16T09:02:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: pgmap v167: 268 pgs: 40 unknown, 228 active+clean; 590 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi111-76430-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi111-76430-27"}]': finished 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"}]': finished 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"}]': finished 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: osdmap e146: 8 total, 8 up, 8 in 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: pgmap v167: 268 pgs: 40 unknown, 228 active+clean; 590 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi111-76430-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi111-76430-27"}]': finished 2024-09-16T09:02:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"}]': finished 2024-09-16T09:02:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"}]': finished 2024-09-16T09:02:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: osdmap e146: 8 total, 8 up, 8 in 2024-09-16T09:02:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: pgmap v167: 268 pgs: 40 unknown, 228 active+clean; 590 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi111-76430-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi111-76430-27"}]': finished 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76617-24"}]': finished 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi111-76517-2"}]': finished 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: osdmap e146: 8 total, 8 up, 8 in 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2206357785' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1446237590' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"} : dispatch 2024-09-16T09:02:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"} : dispatch 2024-09-16T09:02:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:17.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"}]': finished 2024-09-16T09:02:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"}]': finished 2024-09-16T09:02:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:17 smithi114 ceph-mon[33315]: osdmap e147: 8 total, 8 up, 8 in 2024-09-16T09:02:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2520092967' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "StatRemovePP_smithi111-76450-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "StatRemovePP_smithi111-76450-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi111-76450-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:17 smithi114 ceph-mon[33315]: osdmap e148: 8 total, 8 up, 8 in 2024-09-16T09:02:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1085724716' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi111-76617-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi111-76617-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"}]': finished 2024-09-16T09:02:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"}]': finished 2024-09-16T09:02:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[40056]: osdmap e147: 8 total, 8 up, 8 in 2024-09-16T09:02:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2520092967' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "StatRemovePP_smithi111-76450-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "StatRemovePP_smithi111-76450-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi111-76450-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[40056]: osdmap e148: 8 total, 8 up, 8 in 2024-09-16T09:02:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1085724716' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi111-76617-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi111-76617-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi111-76517-2"}]': finished 2024-09-16T09:02:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76617-24"}]': finished 2024-09-16T09:02:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[29803]: osdmap e147: 8 total, 8 up, 8 in 2024-09-16T09:02:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2520092967' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "StatRemovePP_smithi111-76450-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "StatRemovePP_smithi111-76450-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi111-76450-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[29803]: osdmap e148: 8 total, 8 up, 8 in 2024-09-16T09:02:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1085724716' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi111-76617-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:17.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi111-76617-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:18.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: pgmap v170: 276 pgs: 32 creating+peering, 244 active+clean; 603 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.5 KiB/s wr, 2 op/s 2024-09-16T09:02:18.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4174056792' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76517-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76517-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi111-76617-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"}]': finished 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76517-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: osdmap e149: 8 total, 8 up, 8 in 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: pgmap v170: 276 pgs: 32 creating+peering, 244 active+clean; 603 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.5 KiB/s wr, 2 op/s 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4174056792' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76517-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76517-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi111-76617-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"}]': finished 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76517-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: osdmap e149: 8 total, 8 up, 8 in 2024-09-16T09:02:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: pgmap v170: 276 pgs: 32 creating+peering, 244 active+clean; 603 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.5 KiB/s wr, 2 op/s 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4174056792' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76517-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76517-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi111-76617-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi111-76430-27"}]': finished 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76517-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: osdmap e149: 8 total, 8 up, 8 in 2024-09-16T09:02:18.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1023012827' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:18.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"} : dispatch 2024-09-16T09:02:19.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosListEC.Lis api_aio: Running main() from gmock_main.cc 2024-09-16T09:02:19.224 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [----------] Global test environment set-up. 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.TooBig (3064 ms) 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3021 ms) 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2288 ms) 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3010 ms) 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-16T09:02:19.225 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3006 ms) 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3003 ms) 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3009 ms) 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2978 ms) 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3020 ms) 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3028 ms) 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3014 ms) 2024-09-16T09:02:19.226 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3005 ms) 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2961 ms) 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3030 ms) 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.Flush (2946 ms) 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3010 ms) 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3009 ms) 2024-09-16T09:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2978 ms) 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3019 ms) 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2997 ms) 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3022 ms) 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2969 ms) 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3005 ms) 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3002 ms) 2024-09-16T09:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-16T09:02:19.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2996 ms) 2024-09-16T09:02:19.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-16T09:02:19.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3011 ms) 2024-09-16T09:02:19.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [----------] 26 tests from LibRadosAio (77401 ms total) 2024-09-16T09:02:19.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: 2024-09-16T09:02:19.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-16T09:02:19.229 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-16T09:02:19.239 INFO:tasks.workunit.client.0.smithi111.stdout: tObjectsStart 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 1 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 10 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 13 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 7 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 14 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 0 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 15 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 11 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 5 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 8 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 6 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 3 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 4 0 2024-09-16T09:02:19.240 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 12 0 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 9 0 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 2 0 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (40 ms) 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [----------] 3 tests from LibRadosListEC (1097 ms total) 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2993 ms) 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [----------] 1 test from LibRadosListNP (2993 ms total) 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [----------] Global test environment tear-down 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [==========] 11 tests from 3 test suites ran. (84355 ms total) 2024-09-16T09:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout: api_list: [ PASSED ] 11 tests. 2024-09-16T09:02:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4174056792' entity='client.admin' cmd={"prefix":"osd pool rm","pool": "test-rados-api-smithi111-76517-3","pool2":"test-rados-api-smithi111-76517-3","yes_i_really_really_mean_it_not_faking": true} : dispatch 2024-09-16T09:02:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool rm","pool": "test-rados-api-smithi111-76517-3","pool2":"test-rados-api-smithi111-76517-3","yes_i_really_really_mean_it_not_faking": true} : dispatch 2024-09-16T09:02:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"}]': finished 2024-09-16T09:02:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi111-76517-3","pool2":"test-rados-api-smithi111-76517-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-09-16T09:02:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: osdmap e150: 8 total, 8 up, 8 in 2024-09-16T09:02:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:02:19.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/43359464' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi111-76450-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:19.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:02:19.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi111-76450-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:19.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi111-76430-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:19.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi111-76430-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4174056792' entity='client.admin' cmd={"prefix":"osd pool rm","pool": "test-rados-api-smithi111-76517-3","pool2":"test-rados-api-smithi111-76517-3","yes_i_really_really_mean_it_not_faking": true} : dispatch 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool rm","pool": "test-rados-api-smithi111-76517-3","pool2":"test-rados-api-smithi111-76517-3","yes_i_really_really_mean_it_not_faking": true} : dispatch 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"}]': finished 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi111-76517-3","pool2":"test-rados-api-smithi111-76517-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: osdmap e150: 8 total, 8 up, 8 in 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:02:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/43359464' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi111-76450-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi111-76450-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi111-76430-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi111-76430-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4174056792' entity='client.admin' cmd={"prefix":"osd pool rm","pool": "test-rados-api-smithi111-76517-3","pool2":"test-rados-api-smithi111-76517-3","yes_i_really_really_mean_it_not_faking": true} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix":"osd pool rm","pool": "test-rados-api-smithi111-76517-3","pool2":"test-rados-api-smithi111-76517-3","yes_i_really_really_mean_it_not_faking": true} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi111-76430-27"}]': finished 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi111-76517-3","pool2":"test-rados-api-smithi111-76517-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: osdmap e150: 8 total, 8 up, 8 in 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/43359464' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi111-76450-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13"} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi111-76450-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi111-76430-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi111-76430-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:20 smithi114 ceph-mon[33315]: pgmap v173: 300 pgs: 64 unknown, 236 active+clean; 603 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-09-16T09:02:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:20 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13"}]': finished 2024-09-16T09:02:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi111-76450-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi111-76430-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:20 smithi114 ceph-mon[33315]: osdmap e151: 8 total, 8 up, 8 in 2024-09-16T09:02:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "WaitForComplete_smithi111-76430-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "WaitForComplete_smithi111-76430-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087738903' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi111-76617-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[40056]: pgmap v173: 300 pgs: 64 unknown, 236 active+clean; 603 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-09-16T09:02:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13"}]': finished 2024-09-16T09:02:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi111-76450-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi111-76430-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[40056]: osdmap e151: 8 total, 8 up, 8 in 2024-09-16T09:02:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "WaitForComplete_smithi111-76430-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "WaitForComplete_smithi111-76430-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087738903' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi111-76617-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[29803]: pgmap v173: 300 pgs: 64 unknown, 236 active+clean; 603 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-09-16T09:02:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-13"}]': finished 2024-09-16T09:02:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi111-76450-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi111-76430-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[29803]: osdmap e151: 8 total, 8 up, 8 in 2024-09-16T09:02:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "WaitForComplete_smithi111-76430-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "WaitForComplete_smithi111-76430-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087738903' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi111-76617-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:21.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:02:21 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:02:22.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:22 smithi114 ceph-mon[33315]: pgmap v176: 300 pgs: 64 unknown, 236 active+clean; 603 KiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087738903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi111-76617-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:22 smithi114 ceph-mon[33315]: osdmap e152: 8 total, 8 up, 8 in 2024-09-16T09:02:22.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:22 smithi111 ceph-mon[40056]: pgmap v176: 300 pgs: 64 unknown, 236 active+clean; 603 KiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:22.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087738903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi111-76617-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:22.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:22 smithi111 ceph-mon[40056]: osdmap e152: 8 total, 8 up, 8 in 2024-09-16T09:02:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:22 smithi111 ceph-mon[29803]: pgmap v176: 300 pgs: 64 unknown, 236 active+clean; 603 KiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087738903' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi111-76617-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:22.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:22 smithi111 ceph-mon[29803]: osdmap e152: 8 total, 8 up, 8 in 2024-09-16T09:02:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:23 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi111-76430-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi111-76430-28"}]': finished 2024-09-16T09:02:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:23 smithi114 ceph-mon[33315]: osdmap e153: 8 total, 8 up, 8 in 2024-09-16T09:02:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2384979683' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OmapPP_smithi111-76450-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OmapPP_smithi111-76450-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:23 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:23.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:02:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:02:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi111-76430-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi111-76430-28"}]': finished 2024-09-16T09:02:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[29803]: osdmap e153: 8 total, 8 up, 8 in 2024-09-16T09:02:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2384979683' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OmapPP_smithi111-76450-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OmapPP_smithi111-76450-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi111-76430-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi111-76430-28"}]': finished 2024-09-16T09:02:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[40056]: osdmap e153: 8 total, 8 up, 8 in 2024-09-16T09:02:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2384979683' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OmapPP_smithi111-76450-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "OmapPP_smithi111-76450-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:23 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:24 smithi114 ceph-mon[33315]: pgmap v179: 276 pgs: 72 unknown, 204 active+clean; 608 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-16T09:02:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:24 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi111-76450-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:24 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:24 smithi114 ceph-mon[33315]: osdmap e154: 8 total, 8 up, 8 in 2024-09-16T09:02:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1680531712' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi111-76617-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:24 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi111-76617-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[40056]: pgmap v179: 276 pgs: 72 unknown, 204 active+clean; 608 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-16T09:02:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi111-76450-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[40056]: osdmap e154: 8 total, 8 up, 8 in 2024-09-16T09:02:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1680531712' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi111-76617-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi111-76617-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[29803]: pgmap v179: 276 pgs: 72 unknown, 204 active+clean; 608 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-16T09:02:24.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi111-76450-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:24.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:24.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[29803]: osdmap e154: 8 total, 8 up, 8 in 2024-09-16T09:02:24.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1680531712' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi111-76617-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:24.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:24 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi111-76617-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi111-76617-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: osdmap e155: 8 total, 8 up, 8 in 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"}]': finished 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: osdmap e156: 8 total, 8 up, 8 in 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:25.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:25.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1574703042' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWritePP_smithi111-76450-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:25.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWritePP_smithi111-76450-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi111-76617-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: osdmap e155: 8 total, 8 up, 8 in 2024-09-16T09:02:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"}]': finished 2024-09-16T09:02:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: osdmap e156: 8 total, 8 up, 8 in 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1574703042' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWritePP_smithi111-76450-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWritePP_smithi111-76450-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi111-76617-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: osdmap e155: 8 total, 8 up, 8 in 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi111-76430-28"}]': finished 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: osdmap e156: 8 total, 8 up, 8 in 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2646745215' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"} : dispatch 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1574703042' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWritePP_smithi111-76450-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:25.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiWritePP_smithi111-76450-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: pgmap v182: 268 pgs: 64 unknown, 204 active+clean; 608 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-16T09:02:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"}]': finished 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-15"}]': finished 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi111-76450-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: osdmap e157: 8 total, 8 up, 8 in 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-15", "mode": "writeback"} : dispatch 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi111-76617-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-15", "mode": "writeback"} : dispatch 2024-09-16T09:02:26.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi111-76617-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: pgmap v182: 268 pgs: 64 unknown, 204 active+clean; 608 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-16T09:02:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"}]': finished 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-15"}]': finished 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi111-76450-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: osdmap e157: 8 total, 8 up, 8 in 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-15", "mode": "writeback"} : dispatch 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi111-76617-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-15", "mode": "writeback"} : dispatch 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi111-76617-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: pgmap v182: 268 pgs: 64 unknown, 204 active+clean; 608 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi111-76430-28"}]': finished 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-15"}]': finished 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi111-76450-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: osdmap e157: 8 total, 8 up, 8 in 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-15", "mode": "writeback"} : dispatch 2024-09-16T09:02:26.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:26.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi111-76617-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:26.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-15", "mode": "writeback"} : dispatch 2024-09-16T09:02:26.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi111-76617-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:27.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi111-76430-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi111-76430-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"}]': finished 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-15", "mode": "writeback"}]': finished 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi111-76430-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip_smithi111-76430-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: osdmap e158: 8 total, 8 up, 8 in 2024-09-16T09:02:27.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:27.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip_smithi111-76430-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi111-76430-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi111-76430-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"}]': finished 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-15", "mode": "writeback"}]': finished 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi111-76430-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip_smithi111-76430-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: osdmap e158: 8 total, 8 up, 8 in 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip_smithi111-76430-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi111-76430-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi111-76430-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"}]': finished 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-15", "mode": "writeback"}]': finished 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi111-76430-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip_smithi111-76430-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: osdmap e158: 8 total, 8 up, 8 in 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"} : dispatch 2024-09-16T09:02:27.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip_smithi111-76430-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:28.249 INFO:tasks.workunit.client.0.smithi111.stdout:napshotsECPP.SnapRemovePP (2004 ms) 2024-09-16T09:02:28.249 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-16T09:02:28.249 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2007 ms) 2024-09-16T09:02:28.249 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2024 ms) 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9031 ms total) 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4044 ms) 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3958 ms) 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4010 ms) 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12012 ms total) 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: 2024-09-16T09:02:28.250 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-16T09:02:28.251 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (93242 ms total) 2024-09-16T09:02:28.251 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-16T09:02:28.251 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-16T09:02:28.251 INFO:tasks.workunit.client.0.smithi111.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T09:02:28.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:28 smithi114 ceph-mon[33315]: pgmap v185: 260 pgs: 260 active+clean; 584 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:28 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi111-76617-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi111-76617-36"}]': finished 2024-09-16T09:02:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"}]': finished 2024-09-16T09:02:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:28 smithi114 ceph-mon[33315]: osdmap e159: 8 total, 8 up, 8 in 2024-09-16T09:02:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/274472306' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi111-76450-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi111-76450-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[29803]: pgmap v185: 260 pgs: 260 active+clean; 584 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi111-76617-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi111-76617-36"}]': finished 2024-09-16T09:02:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"}]': finished 2024-09-16T09:02:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[29803]: osdmap e159: 8 total, 8 up, 8 in 2024-09-16T09:02:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/274472306' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi111-76450-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi111-76450-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[40056]: pgmap v185: 260 pgs: 260 active+clean; 584 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi111-76617-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi111-76617-36"}]': finished 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/240434004' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi111-76863-21"}]': finished 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[40056]: osdmap e159: 8 total, 8 up, 8 in 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/274472306' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi111-76450-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi111-76450-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:28.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:30.478 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[29803]: pgmap v188: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi111-76430-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi111-76430-29"}]': finished 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi111-76450-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[29803]: osdmap e160: 8 total, 8 up, 8 in 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[40056]: pgmap v188: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi111-76430-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi111-76430-29"}]': finished 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi111-76450-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:30.479 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[40056]: osdmap e160: 8 total, 8 up, 8 in 2024-09-16T09:02:30.480 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:30.480 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:30 smithi114 ceph-mon[33315]: pgmap v188: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi111-76430-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi111-76430-29"}]': finished 2024-09-16T09:02:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi111-76450-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:30 smithi114 ceph-mon[33315]: osdmap e160: 8 total, 8 up, 8 in 2024-09-16T09:02:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15"} : dispatch 2024-09-16T09:02:31.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:31 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:31.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15"}]': finished 2024-09-16T09:02:31.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:31 smithi114 ceph-mon[33315]: osdmap e161: 8 total, 8 up, 8 in 2024-09-16T09:02:31.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:31.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:31.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:02:31.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15"}]': finished 2024-09-16T09:02:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[40056]: osdmap e161: 8 total, 8 up, 8 in 2024-09-16T09:02:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:02:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-15"}]': finished 2024-09-16T09:02:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[29803]: osdmap e161: 8 total, 8 up, 8 in 2024-09-16T09:02:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:02:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:31.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:02:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:02:32.268 INFO:tasks.workunit.client.0.smithi111.stdout: (68325 ms total) 2024-09-16T09:02:32.268 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: 2024-09-16T09:02:32.268 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-16T09:02:32.268 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-16T09:02:32.268 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (107 ms) 2024-09-16T09:02:32.268 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (107 ms total) 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (57 ms) 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (9 ms) 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (66 ms total) 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (39 ms) 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (4 ms) 2024-09-16T09:02:32.269 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (43 ms total) 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (36 ms) 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (4 ms) 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (40 ms total) 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1028 ms) 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1028 ms total) 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: 2024-09-16T09:02:32.270 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-16T09:02:32.271 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (97329 ms total) 2024-09-16T09:02:32.271 INFO:tasks.workunit.client.0.smithi111.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-16T09:02:32.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:32 smithi114 ceph-mon[33315]: pgmap v191: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"}]': finished 2024-09-16T09:02:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:32 smithi114 ceph-mon[33315]: osdmap e162: 8 total, 8 up, 8 in 2024-09-16T09:02:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3439759356' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi111-76450-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi111-76450-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[40056]: pgmap v191: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"}]': finished 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[40056]: osdmap e162: 8 total, 8 up, 8 in 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3439759356' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi111-76450-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi111-76450-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[29803]: pgmap v191: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi111-76617-36"}]': finished 2024-09-16T09:02:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[29803]: osdmap e162: 8 total, 8 up, 8 in 2024-09-16T09:02:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1761267950' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3439759356' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi111-76450-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"} : dispatch 2024-09-16T09:02:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi111-76450-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:33.256 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-16T09:02:33.256 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-16T09:02:33.256 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (3074 ms) 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (17325 ms) 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6013 ms) 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3014 ms) 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3016 ms) 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-16T09:02:33.257 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2959 ms) 2024-09-16T09:02:33.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-16T09:02:33.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3008 ms) 2024-09-16T09:02:33.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-16T09:02:33.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3027 ms) 2024-09-16T09:02:33.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-16T09:02:33.258 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2946 ms) 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3009 ms) 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3008 ms) 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2970 ms) 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3020 ms) 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3009 ms) 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3016 ms) 2024-09-16T09:02:33.259 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2967 ms) 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3000 ms) 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3021 ms) 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2975 ms) 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3023 ms) 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3004 ms) 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-16T09:02:33.260 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2977 ms) 2024-09-16T09:02:33.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-16T09:02:33.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3015 ms) 2024-09-16T09:02:33.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-16T09:02:33.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3006 ms) 2024-09-16T09:02:33.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-16T09:02:33.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3016 ms) 2024-09-16T09:02:33.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-16T09:02:33.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2977 ms) 2024-09-16T09:02:33.261 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-16T09:02:33.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"}]': finished 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi111-76450-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"}]': finished 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: osdmap e163: 8 total, 8 up, 8 in 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"}]': finished 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:33 smithi114 ceph-mon[33315]: osdmap e164: 8 total, 8 up, 8 in 2024-09-16T09:02:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:02:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:02:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"}]': finished 2024-09-16T09:02:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi111-76450-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"}]': finished 2024-09-16T09:02:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: osdmap e163: 8 total, 8 up, 8 in 2024-09-16T09:02:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"}]': finished 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[29803]: osdmap e164: 8 total, 8 up, 8 in 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi111-76617-36"}]': finished 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi111-76450-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi111-76430-29"}]': finished 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2608990350' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: osdmap e163: 8 total, 8 up, 8 in 2024-09-16T09:02:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"} : dispatch 2024-09-16T09:02:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi111-76430-29"}]': finished 2024-09-16T09:02:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:33 smithi111 ceph-mon[40056]: osdmap e164: 8 total, 8 up, 8 in 2024-09-16T09:02:34.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: pgmap v194: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi111-76430-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi111-76430-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi111-76430-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: pgmap v194: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi111-76430-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi111-76430-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi111-76430-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: pgmap v194: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:34.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi111-76430-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:34.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi111-76430-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:34.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:34.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi111-76430-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:35.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:35 smithi114 ceph-mon[33315]: osdmap e165: 8 total, 8 up, 8 in 2024-09-16T09:02:35.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip2_smithi111-76430-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip2_smithi111-76430-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2623112105' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi111-76450-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-17"}]': finished 2024-09-16T09:02:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[29803]: osdmap e165: 8 total, 8 up, 8 in 2024-09-16T09:02:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip2_smithi111-76430-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip2_smithi111-76430-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2623112105' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi111-76450-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-17"}]': finished 2024-09-16T09:02:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[40056]: osdmap e165: 8 total, 8 up, 8 in 2024-09-16T09:02:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip2_smithi111-76430-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTrip2_smithi111-76430-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2623112105' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi111-76450-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-17"}]': finished 2024-09-16T09:02:36.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:36 smithi114 ceph-mon[33315]: pgmap v197: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2623112105' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi111-76450-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-17", "mode": "writeback"} : dispatch 2024-09-16T09:02:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:36 smithi114 ceph-mon[33315]: osdmap e166: 8 total, 8 up, 8 in 2024-09-16T09:02:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-17", "mode": "writeback"} : dispatch 2024-09-16T09:02:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:36 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi111-76430-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi111-76430-30"}]': finished 2024-09-16T09:02:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-17", "mode": "writeback"}]': finished 2024-09-16T09:02:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:36 smithi114 ceph-mon[33315]: osdmap e167: 8 total, 8 up, 8 in 2024-09-16T09:02:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[29803]: pgmap v197: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2623112105' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi111-76450-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-17", "mode": "writeback"} : dispatch 2024-09-16T09:02:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[29803]: osdmap e166: 8 total, 8 up, 8 in 2024-09-16T09:02:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-17", "mode": "writeback"} : dispatch 2024-09-16T09:02:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi111-76430-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi111-76430-30"}]': finished 2024-09-16T09:02:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-17", "mode": "writeback"}]': finished 2024-09-16T09:02:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[29803]: osdmap e167: 8 total, 8 up, 8 in 2024-09-16T09:02:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[40056]: pgmap v197: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2623112105' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi111-76450-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-17", "mode": "writeback"} : dispatch 2024-09-16T09:02:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[40056]: osdmap e166: 8 total, 8 up, 8 in 2024-09-16T09:02:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-17", "mode": "writeback"} : dispatch 2024-09-16T09:02:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi111-76430-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi111-76430-30"}]': finished 2024-09-16T09:02:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-17", "mode": "writeback"}]': finished 2024-09-16T09:02:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:36 smithi111 ceph-mon[40056]: osdmap e167: 8 total, 8 up, 8 in 2024-09-16T09:02:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:37 smithi114 ceph-mon[33315]: osdmap e168: 8 total, 8 up, 8 in 2024-09-16T09:02:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3960838553' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi111-76450-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi111-76450-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[40056]: osdmap e168: 8 total, 8 up, 8 in 2024-09-16T09:02:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3960838553' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi111-76450-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi111-76450-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[29803]: osdmap e168: 8 total, 8 up, 8 in 2024-09-16T09:02:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17"} : dispatch 2024-09-16T09:02:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3960838553' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi111-76450-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi111-76450-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:38.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:38 smithi114 ceph-mon[33315]: pgmap v200: 236 pgs: 8 creating+peering, 228 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:02:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:38 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17"}]': finished 2024-09-16T09:02:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi111-76450-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:38 smithi114 ceph-mon[33315]: osdmap e169: 8 total, 8 up, 8 in 2024-09-16T09:02:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[29803]: pgmap v200: 236 pgs: 8 creating+peering, 228 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:02:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17"}]': finished 2024-09-16T09:02:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi111-76450-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[29803]: osdmap e169: 8 total, 8 up, 8 in 2024-09-16T09:02:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[40056]: pgmap v200: 236 pgs: 8 creating+peering, 228 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:02:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-17"}]': finished 2024-09-16T09:02:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi111-76450-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[40056]: osdmap e169: 8 total, 8 up, 8 in 2024-09-16T09:02:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:40.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:40 smithi114 ceph-mon[33315]: pgmap v203: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:02:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"}]': finished 2024-09-16T09:02:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:40 smithi114 ceph-mon[33315]: osdmap e170: 8 total, 8 up, 8 in 2024-09-16T09:02:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:40 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[29803]: pgmap v203: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:02:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"}]': finished 2024-09-16T09:02:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[29803]: osdmap e170: 8 total, 8 up, 8 in 2024-09-16T09:02:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[40056]: pgmap v203: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:02:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi111-76430-30"}]': finished 2024-09-16T09:02:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[40056]: osdmap e170: 8 total, 8 up, 8 in 2024-09-16T09:02:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1712434567' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"} : dispatch 2024-09-16T09:02:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:40 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:41.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"}]': finished 2024-09-16T09:02:41.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: osdmap e171: 8 total, 8 up, 8 in 2024-09-16T09:02:41.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:41.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2126303595' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi111-76450-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:41.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:41.616 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.617 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.617 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.617 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.617 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi111-76430-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:41.617 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi111-76430-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"}]': finished 2024-09-16T09:02:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: osdmap e171: 8 total, 8 up, 8 in 2024-09-16T09:02:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2126303595' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi111-76450-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi111-76430-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi111-76430-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi111-76430-30"}]': finished 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: osdmap e171: 8 total, 8 up, 8 in 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2126303595' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi111-76450-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:41.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi111-76430-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:41.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi111-76430-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:41.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:02:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:02:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:42 smithi114 ceph-mon[33315]: pgmap v206: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2126303595' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi111-76450-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi111-76430-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:42 smithi114 ceph-mon[33315]: osdmap e172: 8 total, 8 up, 8 in 2024-09-16T09:02:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripAppend_smithi111-76430-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripAppend_smithi111-76430-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[29803]: pgmap v206: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2126303595' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi111-76450-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi111-76430-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[29803]: osdmap e172: 8 total, 8 up, 8 in 2024-09-16T09:02:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripAppend_smithi111-76430-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripAppend_smithi111-76430-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[40056]: pgmap v206: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2126303595' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi111-76450-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi111-76430-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[40056]: osdmap e172: 8 total, 8 up, 8 in 2024-09-16T09:02:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripAppend_smithi111-76430-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripAppend_smithi111-76430-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:43 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:43 smithi114 ceph-mon[33315]: osdmap e173: 8 total, 8 up, 8 in 2024-09-16T09:02:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:02:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:02:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[40056]: osdmap e173: 8 total, 8 up, 8 in 2024-09-16T09:02:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[29803]: osdmap e173: 8 total, 8 up, 8 in 2024-09-16T09:02:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:44.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: pgmap v209: 228 pgs: 228 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:02:44.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi111-76430-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi111-76430-31"}]': finished 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-19"}]': finished 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: osdmap e174: 8 total, 8 up, 8 in 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-19", "mode": "writeback"} : dispatch 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-19", "mode": "writeback"} : dispatch 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2131694623' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi111-76450-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi111-76450-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:02:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:44 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:02:44.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: pgmap v209: 228 pgs: 228 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:02:44.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi111-76430-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi111-76430-31"}]': finished 2024-09-16T09:02:44.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-19"}]': finished 2024-09-16T09:02:44.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: osdmap e174: 8 total, 8 up, 8 in 2024-09-16T09:02:44.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-19", "mode": "writeback"} : dispatch 2024-09-16T09:02:44.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-19", "mode": "writeback"} : dispatch 2024-09-16T09:02:44.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2131694623' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi111-76450-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:44.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi111-76450-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: pgmap v209: 228 pgs: 228 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi111-76430-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi111-76430-31"}]': finished 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-19"}]': finished 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: osdmap e174: 8 total, 8 up, 8 in 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-19", "mode": "writeback"} : dispatch 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-19", "mode": "writeback"} : dispatch 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2131694623' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi111-76450-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi111-76450-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:44.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:02:44.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:02:44.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:02:44.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:44 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:02:45.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:45.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-19", "mode": "writeback"}]': finished 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi111-76450-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: osdmap e175: 8 total, 8 up, 8 in 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2131694623' entity='client.admin' cmd={ 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: "prefix": "osd pool set", 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: "pool": "PoolEIOFlag_smithi111-76450-33", 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: "var": "eio", 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: "val": "true" 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: } : dispatch 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={ 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: "prefix": "osd pool set", 2024-09-16T09:02:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: "pool": "PoolEIOFlag_smithi111-76450-33", 2024-09-16T09:02:45.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: "var": "eio", 2024-09-16T09:02:45.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: "val": "true" 2024-09-16T09:02:45.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:45 smithi114 ceph-mon[33315]: } : dispatch 2024-09-16T09:02:45.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-19", "mode": "writeback"}]': finished 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi111-76450-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: osdmap e175: 8 total, 8 up, 8 in 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2131694623' entity='client.admin' cmd={ 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: "prefix": "osd pool set", 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: "pool": "PoolEIOFlag_smithi111-76450-33", 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: "var": "eio", 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: "val": "true" 2024-09-16T09:02:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: } : dispatch 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={ 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: "prefix": "osd pool set", 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: "pool": "PoolEIOFlag_smithi111-76450-33", 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: "var": "eio", 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: "val": "true" 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[29803]: } : dispatch 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-19", "mode": "writeback"}]': finished 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi111-76450-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: osdmap e175: 8 total, 8 up, 8 in 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:45.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2131694623' entity='client.admin' cmd={ 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: "prefix": "osd pool set", 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: "pool": "PoolEIOFlag_smithi111-76450-33", 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: "var": "eio", 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: "val": "true" 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: } : dispatch 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={ 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: "prefix": "osd pool set", 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: "pool": "PoolEIOFlag_smithi111-76450-33", 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: "var": "eio", 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: "val": "true" 2024-09-16T09:02:45.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:45 smithi111 ceph-mon[40056]: } : dispatch 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: pgmap v212: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{ 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: "prefix": "osd pool set", 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: "pool": "PoolEIOFlag_smithi111-76450-33", 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: "var": "eio", 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: "val": "true" 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: }]': finished 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: osdmap e176: 8 total, 8 up, 8 in 2024-09-16T09:02:46.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:46.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:46.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:02:46.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: pgmap v212: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{ 2024-09-16T09:02:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: "prefix": "osd pool set", 2024-09-16T09:02:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: "pool": "PoolEIOFlag_smithi111-76450-33", 2024-09-16T09:02:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: "var": "eio", 2024-09-16T09:02:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: "val": "true" 2024-09-16T09:02:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: }]': finished 2024-09-16T09:02:46.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:46.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: osdmap e176: 8 total, 8 up, 8 in 2024-09-16T09:02:46.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:46.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:46.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:02:46.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: pgmap v212: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:02:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{ 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: "prefix": "osd pool set", 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: "pool": "PoolEIOFlag_smithi111-76450-33", 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: "var": "eio", 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: "val": "true" 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: }]': finished 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: osdmap e176: 8 total, 8 up, 8 in 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19"} : dispatch 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:46.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:02:46.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:02:47.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:47 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:47 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19"}]': finished 2024-09-16T09:02:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:47 smithi114 ceph-mon[33315]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"}]': finished 2024-09-16T09:02:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:47 smithi114 ceph-mon[33315]: osdmap e177: 8 total, 8 up, 8 in 2024-09-16T09:02:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:47 smithi114 ceph-mon[33315]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:47 smithi114 ceph-mon[33315]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"}]': finished 2024-09-16T09:02:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:47 smithi114 ceph-mon[33315]: osdmap e178: 8 total, 8 up, 8 in 2024-09-16T09:02:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19"}]': finished 2024-09-16T09:02:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[29803]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"}]': finished 2024-09-16T09:02:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[29803]: osdmap e177: 8 total, 8 up, 8 in 2024-09-16T09:02:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[29803]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[29803]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"}]': finished 2024-09-16T09:02:47.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[29803]: osdmap e178: 8 total, 8 up, 8 in 2024-09-16T09:02:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-19"}]': finished 2024-09-16T09:02:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[40056]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi111-76430-31"}]': finished 2024-09-16T09:02:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[40056]: osdmap e177: 8 total, 8 up, 8 in 2024-09-16T09:02:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[40056]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"} : dispatch 2024-09-16T09:02:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[40056]: from='client.42506 172.21.15.111:0/3119663809' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi111-76430-31"}]': finished 2024-09-16T09:02:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:47 smithi111 ceph-mon[40056]: osdmap e178: 8 total, 8 up, 8 in 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: pgmap v215: 228 pgs: 228 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4010763974' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiReads_smithi111-76450-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi111-76430-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi111-76430-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4010763974' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi111-76450-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi111-76430-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:48.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: osdmap e179: 8 total, 8 up, 8 in 2024-09-16T09:02:48.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsComplete_smithi111-76430-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsComplete_smithi111-76430-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:48.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:48 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: pgmap v215: 228 pgs: 228 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:02:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4010763974' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiReads_smithi111-76450-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi111-76430-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi111-76430-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4010763974' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi111-76450-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi111-76430-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: osdmap e179: 8 total, 8 up, 8 in 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsComplete_smithi111-76430-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsComplete_smithi111-76430-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: pgmap v215: 228 pgs: 228 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4010763974' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "MultiReads_smithi111-76450-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi111-76430-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi111-76430-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4010763974' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi111-76450-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi111-76430-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: osdmap e179: 8 total, 8 up, 8 in 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsComplete_smithi111-76430-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsComplete_smithi111-76430-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:48.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:48.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:48 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:50.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:50 smithi114 ceph-mon[33315]: pgmap v218: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:02:50.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:50 smithi114 ceph-mon[33315]: osdmap e180: 8 total, 8 up, 8 in 2024-09-16T09:02:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[29803]: pgmap v218: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:02:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[29803]: osdmap e180: 8 total, 8 up, 8 in 2024-09-16T09:02:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[40056]: pgmap v218: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:02:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[40056]: osdmap e180: 8 total, 8 up, 8 in 2024-09-16T09:02:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi111-76430-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi111-76430-32"}]': finished 2024-09-16T09:02:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:51 smithi114 ceph-mon[33315]: osdmap e181: 8 total, 8 up, 8 in 2024-09-16T09:02:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1383367221' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi111-76450-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi111-76430-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi111-76430-32"}]': finished 2024-09-16T09:02:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[29803]: osdmap e181: 8 total, 8 up, 8 in 2024-09-16T09:02:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1383367221' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi111-76450-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi111-76430-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi111-76430-32"}]': finished 2024-09-16T09:02:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:02:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[40056]: osdmap e181: 8 total, 8 up, 8 in 2024-09-16T09:02:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1383367221' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi111-76450-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:51.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:02:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:02:52.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:52 smithi114 ceph-mon[33315]: pgmap v221: 268 pgs: 72 unknown, 196 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:52 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1383367221' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi111-76450-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-21"}]': finished 2024-09-16T09:02:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-21", "mode": "writeback"} : dispatch 2024-09-16T09:02:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:52 smithi114 ceph-mon[33315]: osdmap e182: 8 total, 8 up, 8 in 2024-09-16T09:02:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-21", "mode": "writeback"} : dispatch 2024-09-16T09:02:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[40056]: pgmap v221: 268 pgs: 72 unknown, 196 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1383367221' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi111-76450-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-21"}]': finished 2024-09-16T09:02:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-21", "mode": "writeback"} : dispatch 2024-09-16T09:02:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[40056]: osdmap e182: 8 total, 8 up, 8 in 2024-09-16T09:02:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-21", "mode": "writeback"} : dispatch 2024-09-16T09:02:52.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[29803]: pgmap v221: 268 pgs: 72 unknown, 196 active+clean; 584 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:02:52.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:52.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1383367221' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi111-76450-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:52.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-21"}]': finished 2024-09-16T09:02:52.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-21", "mode": "writeback"} : dispatch 2024-09-16T09:02:52.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[29803]: osdmap e182: 8 total, 8 up, 8 in 2024-09-16T09:02:52.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-21", "mode": "writeback"} : dispatch 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-21", "mode": "writeback"}]': finished 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: osdmap e183: 8 total, 8 up, 8 in 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"}]': finished 2024-09-16T09:02:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:53.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:53.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: osdmap e184: 8 total, 8 up, 8 in 2024-09-16T09:02:53.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2228407132' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:53.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:53.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:02:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:02:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-21", "mode": "writeback"}]': finished 2024-09-16T09:02:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: osdmap e183: 8 total, 8 up, 8 in 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"}]': finished 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: osdmap e184: 8 total, 8 up, 8 in 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2228407132' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-21", "mode": "writeback"}]': finished 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: osdmap e183: 8 total, 8 up, 8 in 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi111-76430-32"}]': finished 2024-09-16T09:02:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:02:53.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:53.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: osdmap e184: 8 total, 8 up, 8 in 2024-09-16T09:02:53.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3860988106' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2228407132' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:53.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21"} : dispatch 2024-09-16T09:02:53.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"} : dispatch 2024-09-16T09:02:53.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:54 smithi114 ceph-mon[33315]: pgmap v224: 228 pgs: 228 active+clean; 584 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:02:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:54 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:54 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21"}]': finished 2024-09-16T09:02:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:54 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"}]': finished 2024-09-16T09:02:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:54 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:54 smithi114 ceph-mon[33315]: osdmap e185: 8 total, 8 up, 8 in 2024-09-16T09:02:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[29803]: pgmap v224: 228 pgs: 228 active+clean; 584 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:02:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:54.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21"}]': finished 2024-09-16T09:02:54.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"}]': finished 2024-09-16T09:02:54.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:54.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[29803]: osdmap e185: 8 total, 8 up, 8 in 2024-09-16T09:02:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[40056]: pgmap v224: 228 pgs: 228 active+clean; 584 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:02:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:02:54.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-21"}]': finished 2024-09-16T09:02:54.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi111-76430-32"}]': finished 2024-09-16T09:02:54.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:54.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:54 smithi111 ceph-mon[40056]: osdmap e185: 8 total, 8 up, 8 in 2024-09-16T09:02:55.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:02:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:02:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi111-76430-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi111-76430-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:55 smithi114 ceph-mon[33315]: osdmap e186: 8 total, 8 up, 8 in 2024-09-16T09:02:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsSafe_smithi111-76430-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:02:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:02:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:02:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi111-76430-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi111-76430-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[29803]: osdmap e186: 8 total, 8 up, 8 in 2024-09-16T09:02:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsSafe_smithi111-76430-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:02:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:02:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:02:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi111-76430-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:02:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi111-76430-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:02:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[40056]: osdmap e186: 8 total, 8 up, 8 in 2024-09-16T09:02:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsSafe_smithi111-76430-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:02:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:56.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:56 smithi114 ceph-mon[33315]: pgmap v227: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:02:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3057712' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:56 smithi114 ceph-mon[33315]: osdmap e187: 8 total, 8 up, 8 in 2024-09-16T09:02:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[29803]: pgmap v227: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:02:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3057712' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[29803]: osdmap e187: 8 total, 8 up, 8 in 2024-09-16T09:02:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[40056]: pgmap v227: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:02:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3057712' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[40056]: osdmap e187: 8 total, 8 up, 8 in 2024-09-16T09:02:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:58 smithi114 ceph-mon[33315]: pgmap v230: 292 pgs: 32 creating+peering, 260 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:02:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi111-76430-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi111-76430-33"}]': finished 2024-09-16T09:02:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:58 smithi114 ceph-mon[33315]: osdmap e188: 8 total, 8 up, 8 in 2024-09-16T09:02:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/5363760' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:58 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[29803]: pgmap v230: 292 pgs: 32 creating+peering, 260 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:02:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi111-76430-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi111-76430-33"}]': finished 2024-09-16T09:02:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[29803]: osdmap e188: 8 total, 8 up, 8 in 2024-09-16T09:02:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/5363760' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:58.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[40056]: pgmap v230: 292 pgs: 32 creating+peering, 260 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:02:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi111-76430-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi111-76430-33"}]': finished 2024-09-16T09:02:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[40056]: osdmap e188: 8 total, 8 up, 8 in 2024-09-16T09:02:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/5363760' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:02:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:58 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:02:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:59 smithi114 ceph-mon[33315]: osdmap e189: 8 total, 8 up, 8 in 2024-09-16T09:02:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:02:59 smithi114 ceph-mon[33315]: pgmap v233: 332 pgs: 40 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:02:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[29803]: osdmap e189: 8 total, 8 up, 8 in 2024-09-16T09:02:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[29803]: pgmap v233: 332 pgs: 40 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:02:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi111-76450-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:02:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[40056]: osdmap e189: 8 total, 8 up, 8 in 2024-09-16T09:02:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:02:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:02:59 smithi111 ceph-mon[40056]: pgmap v233: 332 pgs: 40 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:03:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:00 smithi114 ceph-mon[33315]: osdmap e190: 8 total, 8 up, 8 in 2024-09-16T09:03:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:03:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[29803]: osdmap e190: 8 total, 8 up, 8 in 2024-09-16T09:03:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:03:00.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[40056]: osdmap e190: 8 total, 8 up, 8 in 2024-09-16T09:03:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-23"}]': finished 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi111-76430-33"}]': finished 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: osdmap e191: 8 total, 8 up, 8 in 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-23", "mode": "writeback"} : dispatch 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-23", "mode": "writeback"} : dispatch 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: pgmap v236: 260 pgs: 32 creating+peering, 228 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:01.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi111-76430-33"}]': finished 2024-09-16T09:03:01.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-23", "mode": "writeback"}]': finished 2024-09-16T09:03:01.636 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-mon[33315]: osdmap e192: 8 total, 8 up, 8 in 2024-09-16T09:03:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-23"}]': finished 2024-09-16T09:03:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi111-76430-33"}]': finished 2024-09-16T09:03:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: osdmap e191: 8 total, 8 up, 8 in 2024-09-16T09:03:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-23", "mode": "writeback"} : dispatch 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-23", "mode": "writeback"} : dispatch 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: pgmap v236: 260 pgs: 32 creating+peering, 228 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi111-76430-33"}]': finished 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-23", "mode": "writeback"}]': finished 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[29803]: osdmap e192: 8 total, 8 up, 8 in 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-23"}]': finished 2024-09-16T09:03:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi111-76430-33"}]': finished 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: osdmap e191: 8 total, 8 up, 8 in 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafe_smithi111-76430-33"} : dispatch 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-23", "mode": "writeback"} : dispatch 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-23", "mode": "writeback"} : dispatch 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: pgmap v236: 260 pgs: 32 creating+peering, 228 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1511063134' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi111-76430-33"}]': finished 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-23", "mode": "writeback"}]': finished 2024-09-16T09:03:01.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:01 smithi111 ceph-mon[40056]: osdmap e192: 8 total, 8 up, 8 in 2024-09-16T09:03:01.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:03:01 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:03:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi111-76430-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi111-76430-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi111-76430-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi111-76430-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:03 smithi114 ceph-mon[33315]: osdmap e193: 8 total, 8 up, 8 in 2024-09-16T09:03:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReturnValue_smithi111-76430-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/454201472' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:03 smithi114 ceph-mon[33315]: pgmap v239: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 542 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-16T09:03:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:03 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:03:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:03:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi111-76430-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[40056]: osdmap e193: 8 total, 8 up, 8 in 2024-09-16T09:03:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReturnValue_smithi111-76430-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/454201472' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[40056]: pgmap v239: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 542 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi111-76430-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[29803]: osdmap e193: 8 total, 8 up, 8 in 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReturnValue_smithi111-76430-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/454201472' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:03.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[29803]: pgmap v239: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 542 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-16T09:03:03.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:03 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:04 smithi114 ceph-mon[33315]: osdmap e194: 8 total, 8 up, 8 in 2024-09-16T09:03:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[29803]: osdmap e194: 8 total, 8 up, 8 in 2024-09-16T09:03:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[40056]: osdmap e194: 8 total, 8 up, 8 in 2024-09-16T09:03:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23"} : dispatch 2024-09-16T09:03:05.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:05 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi111-76430-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi111-76430-34"}]': finished 2024-09-16T09:03:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23"}]': finished 2024-09-16T09:03:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:05 smithi114 ceph-mon[33315]: osdmap e195: 8 total, 8 up, 8 in 2024-09-16T09:03:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/635934586' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:05 smithi114 ceph-mon[33315]: pgmap v242: 300 pgs: 72 unknown, 228 active+clean; 584 KiB data, 542 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-16T09:03:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/635934586' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:05 smithi114 ceph-mon[33315]: osdmap e196: 8 total, 8 up, 8 in 2024-09-16T09:03:05.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:05.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi111-76430-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi111-76430-34"}]': finished 2024-09-16T09:03:05.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23"}]': finished 2024-09-16T09:03:05.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[40056]: osdmap e195: 8 total, 8 up, 8 in 2024-09-16T09:03:05.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/635934586' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:05.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[40056]: pgmap v242: 300 pgs: 72 unknown, 228 active+clean; 584 KiB data, 542 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/635934586' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[40056]: osdmap e196: 8 total, 8 up, 8 in 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi111-76430-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi111-76430-34"}]': finished 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-23"}]': finished 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[29803]: osdmap e195: 8 total, 8 up, 8 in 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/635934586' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[29803]: pgmap v242: 300 pgs: 72 unknown, 228 active+clean; 584 KiB data, 542 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/635934586' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:05 smithi111 ceph-mon[29803]: osdmap e196: 8 total, 8 up, 8 in 2024-09-16T09:03:07.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:07 smithi114 ceph-mon[33315]: osdmap e197: 8 total, 8 up, 8 in 2024-09-16T09:03:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2832228426' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:07 smithi114 ceph-mon[33315]: pgmap v245: 324 pgs: 64 creating+peering, 260 active+clean; 584 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[40056]: osdmap e197: 8 total, 8 up, 8 in 2024-09-16T09:03:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2832228426' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[40056]: pgmap v245: 324 pgs: 64 creating+peering, 260 active+clean; 584 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[29803]: osdmap e197: 8 total, 8 up, 8 in 2024-09-16T09:03:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2832228426' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:07.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:07 smithi111 ceph-mon[29803]: pgmap v245: 324 pgs: 64 creating+peering, 260 active+clean; 584 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi111-76430-34"}]': finished 2024-09-16T09:03:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2832228426' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: osdmap e198: 8 total, 8 up, 8 in 2024-09-16T09:03:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi111-76430-34"}]': finished 2024-09-16T09:03:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: osdmap e199: 8 total, 8 up, 8 in 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1837633181' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi111-76430-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi111-76430-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi111-76430-34"}]': finished 2024-09-16T09:03:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2832228426' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: osdmap e198: 8 total, 8 up, 8 in 2024-09-16T09:03:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi111-76430-34"}]': finished 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: osdmap e199: 8 total, 8 up, 8 in 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1837633181' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi111-76430-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi111-76430-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi111-76430-34"}]': finished 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2832228426' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: osdmap e198: 8 total, 8 up, 8 in 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValue_smithi111-76430-34"} : dispatch 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4085508418' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi111-76430-34"}]': finished 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: osdmap e199: 8 total, 8 up, 8 in 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1837633181' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi111-76430-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:08.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi111-76430-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:10.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:10 smithi114 ceph-mon[33315]: pgmap v248: 356 pgs: 32 unknown, 64 creating+peering, 260 active+clean; 584 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi111-76430-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:10 smithi114 ceph-mon[33315]: osdmap e200: 8 total, 8 up, 8 in 2024-09-16T09:03:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "Flush_smithi111-76430-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "Flush_smithi111-76430-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[29803]: pgmap v248: 356 pgs: 32 unknown, 64 creating+peering, 260 active+clean; 584 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi111-76430-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[29803]: osdmap e200: 8 total, 8 up, 8 in 2024-09-16T09:03:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "Flush_smithi111-76430-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "Flush_smithi111-76430-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[40056]: pgmap v248: 356 pgs: 32 unknown, 64 creating+peering, 260 active+clean; 584 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi111-76430-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[40056]: osdmap e200: 8 total, 8 up, 8 in 2024-09-16T09:03:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "Flush_smithi111-76430-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "Flush_smithi111-76430-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:11.639 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-25"}]': finished 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-25", "mode": "writeback"} : dispatch 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: osdmap e201: 8 total, 8 up, 8 in 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3529050088' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-25", "mode": "writeback"} : dispatch 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi111-76430-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi111-76430-35"}]': finished 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-25", "mode": "writeback"}]': finished 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:11.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-mon[33315]: osdmap e202: 8 total, 8 up, 8 in 2024-09-16T09:03:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-25"}]': finished 2024-09-16T09:03:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-25", "mode": "writeback"} : dispatch 2024-09-16T09:03:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: osdmap e201: 8 total, 8 up, 8 in 2024-09-16T09:03:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3529050088' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-25", "mode": "writeback"} : dispatch 2024-09-16T09:03:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi111-76430-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi111-76430-35"}]': finished 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-25", "mode": "writeback"}]': finished 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[40056]: osdmap e202: 8 total, 8 up, 8 in 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-25"}]': finished 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-25", "mode": "writeback"} : dispatch 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: osdmap e201: 8 total, 8 up, 8 in 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3529050088' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-25", "mode": "writeback"} : dispatch 2024-09-16T09:03:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:11.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:11.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi111-76430-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi111-76430-35"}]': finished 2024-09-16T09:03:11.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-25", "mode": "writeback"}]': finished 2024-09-16T09:03:11.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi111-76450-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:11.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:11 smithi111 ceph-mon[29803]: osdmap e202: 8 total, 8 up, 8 in 2024-09-16T09:03:11.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:03:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:03:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:12 smithi114 ceph-mon[33315]: pgmap v251: 388 pgs: 64 unknown, 64 creating+peering, 260 active+clean; 584 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:12 smithi114 ceph-mon[33315]: osdmap e203: 8 total, 8 up, 8 in 2024-09-16T09:03:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[40056]: pgmap v251: 388 pgs: 64 unknown, 64 creating+peering, 260 active+clean; 584 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[40056]: osdmap e203: 8 total, 8 up, 8 in 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[29803]: pgmap v251: 388 pgs: 64 unknown, 64 creating+peering, 260 active+clean; 584 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[29803]: osdmap e203: 8 total, 8 up, 8 in 2024-09-16T09:03:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25"} : dispatch 2024-09-16T09:03:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:13 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:13 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25"}]': finished 2024-09-16T09:03:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:13 smithi114 ceph-mon[33315]: osdmap e204: 8 total, 8 up, 8 in 2024-09-16T09:03:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:13.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:03:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25"}]': finished 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[40056]: osdmap e204: 8 total, 8 up, 8 in 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:13.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-25"}]': finished 2024-09-16T09:03:13.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[29803]: osdmap e204: 8 total, 8 up, 8 in 2024-09-16T09:03:13.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:13.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:14.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:14 smithi114 ceph-mon[33315]: pgmap v254: 364 pgs: 8 creating+peering, 356 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:03:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"}]': finished 2024-09-16T09:03:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:14 smithi114 ceph-mon[33315]: osdmap e205: 8 total, 8 up, 8 in 2024-09-16T09:03:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[29803]: pgmap v254: 364 pgs: 8 creating+peering, 356 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:03:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"}]': finished 2024-09-16T09:03:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[29803]: osdmap e205: 8 total, 8 up, 8 in 2024-09-16T09:03:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[40056]: pgmap v254: 364 pgs: 8 creating+peering, 356 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:03:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi111-76430-35"}]': finished 2024-09-16T09:03:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[40056]: osdmap e205: 8 total, 8 up, 8 in 2024-09-16T09:03:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2786686406' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"} : dispatch 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: pgmap v257: 260 pgs: 260 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"}]': finished 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: osdmap e206: 8 total, 8 up, 8 in 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi111-76430-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi111-76430-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: pgmap v257: 260 pgs: 260 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"}]': finished 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: osdmap e206: 8 total, 8 up, 8 in 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi111-76430-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi111-76430-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: pgmap v257: 260 pgs: 260 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi111-76430-35"}]': finished 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: osdmap e206: 8 total, 8 up, 8 in 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi111-76430-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi111-76430-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:17.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi111-76430-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:17 smithi114 ceph-mon[33315]: osdmap e207: 8 total, 8 up, 8 in 2024-09-16T09:03:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushAsync_smithi111-76430-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushAsync_smithi111-76430-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi111-76430-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[29803]: osdmap e207: 8 total, 8 up, 8 in 2024-09-16T09:03:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushAsync_smithi111-76430-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushAsync_smithi111-76430-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi111-76430-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[40056]: osdmap e207: 8 total, 8 up, 8 in 2024-09-16T09:03:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushAsync_smithi111-76430-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushAsync_smithi111-76430-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:18 smithi114 ceph-mon[33315]: pgmap v260: 228 pgs: 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-16T09:03:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:18 smithi114 ceph-mon[33315]: osdmap e208: 8 total, 8 up, 8 in 2024-09-16T09:03:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2949178690' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi111-76450-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:18 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[29803]: pgmap v260: 228 pgs: 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-16T09:03:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[29803]: osdmap e208: 8 total, 8 up, 8 in 2024-09-16T09:03:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2949178690' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi111-76450-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:18.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[40056]: pgmap v260: 228 pgs: 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-16T09:03:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[40056]: osdmap e208: 8 total, 8 up, 8 in 2024-09-16T09:03:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2949178690' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi111-76450-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:18 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:19.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi111-76430-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi111-76430-36"}]': finished 2024-09-16T09:03:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2949178690' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi111-76450-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:19 smithi114 ceph-mon[33315]: osdmap e209: 8 total, 8 up, 8 in 2024-09-16T09:03:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi111-76430-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi111-76430-36"}]': finished 2024-09-16T09:03:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2949178690' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi111-76450-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[29803]: osdmap e209: 8 total, 8 up, 8 in 2024-09-16T09:03:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi111-76430-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi111-76430-36"}]': finished 2024-09-16T09:03:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2949178690' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi111-76450-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[40056]: osdmap e209: 8 total, 8 up, 8 in 2024-09-16T09:03:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:20 smithi114 ceph-mon[33315]: pgmap v263: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:03:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-27"}]': finished 2024-09-16T09:03:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-27", "mode": "writeback"} : dispatch 2024-09-16T09:03:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:20 smithi114 ceph-mon[33315]: osdmap e210: 8 total, 8 up, 8 in 2024-09-16T09:03:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-27", "mode": "writeback"} : dispatch 2024-09-16T09:03:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[29803]: pgmap v263: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:03:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-27"}]': finished 2024-09-16T09:03:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-27", "mode": "writeback"} : dispatch 2024-09-16T09:03:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[29803]: osdmap e210: 8 total, 8 up, 8 in 2024-09-16T09:03:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-27", "mode": "writeback"} : dispatch 2024-09-16T09:03:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[40056]: pgmap v263: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:03:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-27"}]': finished 2024-09-16T09:03:20.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-27", "mode": "writeback"} : dispatch 2024-09-16T09:03:20.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[40056]: osdmap e210: 8 total, 8 up, 8 in 2024-09-16T09:03:20.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-27", "mode": "writeback"} : dispatch 2024-09-16T09:03:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-27", "mode": "writeback"}]': finished 2024-09-16T09:03:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: osdmap e211: 8 total, 8 up, 8 in 2024-09-16T09:03:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3055239294' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrListPP_smithi111-76450-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrListPP_smithi111-76450-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"}]': finished 2024-09-16T09:03:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi111-76450-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:21.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: osdmap e212: 8 total, 8 up, 8 in 2024-09-16T09:03:21.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-27", "mode": "writeback"}]': finished 2024-09-16T09:03:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: osdmap e211: 8 total, 8 up, 8 in 2024-09-16T09:03:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3055239294' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrListPP_smithi111-76450-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrListPP_smithi111-76450-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"}]': finished 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi111-76450-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: osdmap e212: 8 total, 8 up, 8 in 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-27", "mode": "writeback"}]': finished 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: osdmap e211: 8 total, 8 up, 8 in 2024-09-16T09:03:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3055239294' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrListPP_smithi111-76450-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:21.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "XattrListPP_smithi111-76450-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:21.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi111-76430-36"}]': finished 2024-09-16T09:03:21.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi111-76450-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:21.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: osdmap e212: 8 total, 8 up, 8 in 2024-09-16T09:03:21.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2701384750' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"} : dispatch 2024-09-16T09:03:21.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:03:21 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:03:22.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:22 smithi114 ceph-mon[33315]: pgmap v266: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"}]': finished 2024-09-16T09:03:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:22 smithi114 ceph-mon[33315]: osdmap e213: 8 total, 8 up, 8 in 2024-09-16T09:03:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[29803]: pgmap v266: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"}]': finished 2024-09-16T09:03:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[29803]: osdmap e213: 8 total, 8 up, 8 in 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[40056]: pgmap v266: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi111-76430-36"}]': finished 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[40056]: osdmap e213: 8 total, 8 up, 8 in 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27"} : dispatch 2024-09-16T09:03:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27"}]': finished 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: osdmap e214: 8 total, 8 up, 8 in 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi111-76430-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:23.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:03:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:03:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27"}]': finished 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: osdmap e214: 8 total, 8 up, 8 in 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi111-76430-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:23.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:23.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-27"}]': finished 2024-09-16T09:03:23.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:23.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:23.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: osdmap e214: 8 total, 8 up, 8 in 2024-09-16T09:03:23.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:23.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi111-76430-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:23.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:24.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:24 smithi114 ceph-mon[33315]: pgmap v269: 228 pgs: 228 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:03:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:24 smithi114 ceph-mon[33315]: osdmap e215: 8 total, 8 up, 8 in 2024-09-16T09:03:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:24 smithi111 ceph-mon[29803]: pgmap v269: 228 pgs: 228 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:03:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:24 smithi111 ceph-mon[29803]: osdmap e215: 8 total, 8 up, 8 in 2024-09-16T09:03:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:24 smithi111 ceph-mon[40056]: pgmap v269: 228 pgs: 228 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:03:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:24 smithi111 ceph-mon[40056]: osdmap e215: 8 total, 8 up, 8 in 2024-09-16T09:03:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:26 smithi114 ceph-mon[33315]: pgmap v272: 196 pgs: 196 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:03:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi111-76430-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi111-76430-37"}]': finished 2024-09-16T09:03:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-46"}]': finished 2024-09-16T09:03:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:26 smithi114 ceph-mon[33315]: osdmap e216: 8 total, 8 up, 8 in 2024-09-16T09:03:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[29803]: pgmap v272: 196 pgs: 196 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:03:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi111-76430-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi111-76430-37"}]': finished 2024-09-16T09:03:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-46"}]': finished 2024-09-16T09:03:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[29803]: osdmap e216: 8 total, 8 up, 8 in 2024-09-16T09:03:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:26.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[40056]: pgmap v272: 196 pgs: 196 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:03:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi111-76430-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi111-76430-37"}]': finished 2024-09-16T09:03:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-46"}]': finished 2024-09-16T09:03:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[40056]: osdmap e216: 8 total, 8 up, 8 in 2024-09-16T09:03:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:27 smithi114 ceph-mon[33315]: osdmap e217: 8 total, 8 up, 8 in 2024-09-16T09:03:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:27 smithi114 ceph-mon[33315]: osdmap e218: 8 total, 8 up, 8 in 2024-09-16T09:03:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[29803]: osdmap e217: 8 total, 8 up, 8 in 2024-09-16T09:03:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[29803]: osdmap e218: 8 total, 8 up, 8 in 2024-09-16T09:03:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[40056]: osdmap e217: 8 total, 8 up, 8 in 2024-09-16T09:03:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[40056]: osdmap e218: 8 total, 8 up, 8 in 2024-09-16T09:03:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: pgmap v275: 244 pgs: 244 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.0 MiB/s wr, 1 op/s 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37"}]': finished 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"}]': finished 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: osdmap e219: 8 total, 8 up, 8 in 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: pgmap v275: 244 pgs: 244 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.0 MiB/s wr, 1 op/s 2024-09-16T09:03:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37"}]': finished 2024-09-16T09:03:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"}]': finished 2024-09-16T09:03:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:28.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: osdmap e219: 8 total, 8 up, 8 in 2024-09-16T09:03:28.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:28.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:28.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:28.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: pgmap v275: 244 pgs: 244 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.0 MiB/s wr, 1 op/s 2024-09-16T09:03:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi111-76430-37"}]': finished 2024-09-16T09:03:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-46"}]': finished 2024-09-16T09:03:28.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1422186820' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:28.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: osdmap e219: 8 total, 8 up, 8 in 2024-09-16T09:03:28.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi111-76430-37"} : dispatch 2024-09-16T09:03:28.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"} : dispatch 2024-09-16T09:03:28.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:28.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: pgmap v278: 228 pgs: 228 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-16T09:03:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi111-76430-37"}]': finished 2024-09-16T09:03:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"}]': finished 2024-09-16T09:03:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: osdmap e220: 8 total, 8 up, 8 in 2024-09-16T09:03:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:30.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:30.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:30.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:30.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi111-76430-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: pgmap v278: 228 pgs: 228 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-16T09:03:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi111-76430-37"}]': finished 2024-09-16T09:03:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"}]': finished 2024-09-16T09:03:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: osdmap e220: 8 total, 8 up, 8 in 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi111-76430-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: pgmap v278: 228 pgs: 228 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2558944807' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi111-76430-37"}]': finished 2024-09-16T09:03:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-46"}]': finished 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: osdmap e220: 8 total, 8 up, 8 in 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:30.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi111-76430-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:31.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-29"}]': finished 2024-09-16T09:03:31.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:31.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi111-76430-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:31.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: osdmap e221: 8 total, 8 up, 8 in 2024-09-16T09:03:31.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-29", "mode": "writeback"} : dispatch 2024-09-16T09:03:31.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:31.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStat_smithi111-76430-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:31.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-29", "mode": "writeback"} : dispatch 2024-09-16T09:03:31.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "107.2", "id": [3, 7]} : dispatch 2024-09-16T09:03:31.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:31.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:31.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-29", "mode": "writeback"}]': finished 2024-09-16T09:03:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-29"}]': finished 2024-09-16T09:03:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi111-76430-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: osdmap e221: 8 total, 8 up, 8 in 2024-09-16T09:03:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-29", "mode": "writeback"} : dispatch 2024-09-16T09:03:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStat_smithi111-76430-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-29", "mode": "writeback"} : dispatch 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "107.2", "id": [3, 7]} : dispatch 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-29", "mode": "writeback"}]': finished 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-29"}]': finished 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi111-76450-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi111-76430-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: osdmap e221: 8 total, 8 up, 8 in 2024-09-16T09:03:31.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-29", "mode": "writeback"} : dispatch 2024-09-16T09:03:31.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:31.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStat_smithi111-76430-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:31.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-29", "mode": "writeback"} : dispatch 2024-09-16T09:03:31.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "107.2", "id": [3, 7]} : dispatch 2024-09-16T09:03:31.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:31.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:31.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-29", "mode": "writeback"}]': finished 2024-09-16T09:03:31.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:03:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:03:32.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:32 smithi114 ceph-mon[33315]: pgmap v281: 228 pgs: 228 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "107.2", "id": [3, 7]}]': finished 2024-09-16T09:03:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:32 smithi114 ceph-mon[33315]: osdmap e222: 8 total, 8 up, 8 in 2024-09-16T09:03:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-47"}]': finished 2024-09-16T09:03:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi111-76430-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi111-76430-38"}]': finished 2024-09-16T09:03:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:32 smithi114 ceph-mon[33315]: osdmap e223: 8 total, 8 up, 8 in 2024-09-16T09:03:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[29803]: pgmap v281: 228 pgs: 228 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "107.2", "id": [3, 7]}]': finished 2024-09-16T09:03:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[29803]: osdmap e222: 8 total, 8 up, 8 in 2024-09-16T09:03:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-47"}]': finished 2024-09-16T09:03:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi111-76430-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi111-76430-38"}]': finished 2024-09-16T09:03:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[29803]: osdmap e223: 8 total, 8 up, 8 in 2024-09-16T09:03:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[40056]: pgmap v281: 228 pgs: 228 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "107.2", "id": [3, 7]}]': finished 2024-09-16T09:03:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[40056]: osdmap e222: 8 total, 8 up, 8 in 2024-09-16T09:03:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi111-76450-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi111-76450-47"}]': finished 2024-09-16T09:03:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi111-76430-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi111-76430-38"}]': finished 2024-09-16T09:03:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:32 smithi111 ceph-mon[40056]: osdmap e223: 8 total, 8 up, 8 in 2024-09-16T09:03:33.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:33 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:33 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:33 smithi114 ceph-mon[33315]: osdmap e224: 8 total, 8 up, 8 in 2024-09-16T09:03:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:33 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:33 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:33 smithi111 ceph-mon[29803]: osdmap e224: 8 total, 8 up, 8 in 2024-09-16T09:03:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:03:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:03:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:03:33.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:33 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:33 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:33 smithi111 ceph-mon[40056]: osdmap e224: 8 total, 8 up, 8 in 2024-09-16T09:03:34.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:34 smithi114 ceph-mon[33315]: pgmap v284: 244 pgs: 16 unknown, 1 peering, 227 active+clean; 13 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T09:03:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:34 smithi114 ceph-mon[33315]: osdmap e225: 8 total, 8 up, 8 in 2024-09-16T09:03:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:34 smithi111 ceph-mon[29803]: pgmap v284: 244 pgs: 16 unknown, 1 peering, 227 active+clean; 13 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T09:03:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:34 smithi111 ceph-mon[29803]: osdmap e225: 8 total, 8 up, 8 in 2024-09-16T09:03:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:34.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:34 smithi111 ceph-mon[40056]: pgmap v284: 244 pgs: 16 unknown, 1 peering, 227 active+clean; 13 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T09:03:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:34 smithi111 ceph-mon[40056]: osdmap e225: 8 total, 8 up, 8 in 2024-09-16T09:03:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi111-76430-38"}]': finished 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-47"}]': finished 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: osdmap e226: 8 total, 8 up, 8 in 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi111-76430-38"}]': finished 2024-09-16T09:03:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-47"}]': finished 2024-09-16T09:03:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: osdmap e226: 8 total, 8 up, 8 in 2024-09-16T09:03:35.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi111-76430-38"}]': finished 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi111-76450-47"}]': finished 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: osdmap e226: 8 total, 8 up, 8 in 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-47"} : dispatch 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:35.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStat_smithi111-76430-38"} : dispatch 2024-09-16T09:03:35.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29"} : dispatch 2024-09-16T09:03:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: pgmap v287: 228 pgs: 1 peering, 227 active+clean; 13 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T09:03:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-47"}]': finished 2024-09-16T09:03:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi111-76430-38"}]': finished 2024-09-16T09:03:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29"}]': finished 2024-09-16T09:03:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: osdmap e227: 8 total, 8 up, 8 in 2024-09-16T09:03:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:36.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi111-76450-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:36.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: pgmap v287: 228 pgs: 1 peering, 227 active+clean; 13 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T09:03:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-47"}]': finished 2024-09-16T09:03:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi111-76430-38"}]': finished 2024-09-16T09:03:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29"}]': finished 2024-09-16T09:03:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: osdmap e227: 8 total, 8 up, 8 in 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi111-76450-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: pgmap v287: 228 pgs: 1 peering, 227 active+clean; 13 MiB data, 566 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3639501833' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi111-76450-47"}]': finished 2024-09-16T09:03:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/368727164' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi111-76430-38"}]': finished 2024-09-16T09:03:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-29"}]': finished 2024-09-16T09:03:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: osdmap e227: 8 total, 8 up, 8 in 2024-09-16T09:03:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi111-76450-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi111-76430-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi111-76430-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi111-76450-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi111-76430-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:37 smithi114 ceph-mon[33315]: osdmap e228: 8 total, 8 up, 8 in 2024-09-16T09:03:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatNS_smithi111-76430-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "WaitForSafePP_smithi111-76450-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatNS_smithi111-76430-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:37.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi111-76430-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:37.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi111-76430-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:37.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi111-76450-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:37.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi111-76430-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:37.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[29803]: osdmap e228: 8 total, 8 up, 8 in 2024-09-16T09:03:37.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatNS_smithi111-76430-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:37.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "WaitForSafePP_smithi111-76450-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:37.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatNS_smithi111-76430-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:37.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi111-76430-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:37.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi111-76430-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:37.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi111-76450-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:37.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi111-76430-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:37.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[40056]: osdmap e228: 8 total, 8 up, 8 in 2024-09-16T09:03:37.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatNS_smithi111-76430-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:37.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "WaitForSafePP_smithi111-76450-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:37.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatNS_smithi111-76430-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:38.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:38 smithi114 ceph-mon[33315]: pgmap v290: 228 pgs: 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 1024 KiB/s wr, 22 op/s; 3 B/s, 1 objects/s recovering 2024-09-16T09:03:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:38 smithi114 ceph-mon[33315]: osdmap e229: 8 total, 8 up, 8 in 2024-09-16T09:03:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:38 smithi111 ceph-mon[29803]: pgmap v290: 228 pgs: 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 1024 KiB/s wr, 22 op/s; 3 B/s, 1 objects/s recovering 2024-09-16T09:03:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:38 smithi111 ceph-mon[29803]: osdmap e229: 8 total, 8 up, 8 in 2024-09-16T09:03:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:38 smithi111 ceph-mon[40056]: pgmap v290: 228 pgs: 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 1024 KiB/s wr, 22 op/s; 3 B/s, 1 objects/s recovering 2024-09-16T09:03:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:38 smithi111 ceph-mon[40056]: osdmap e229: 8 total, 8 up, 8 in 2024-09-16T09:03:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:40 smithi114 ceph-mon[33315]: pgmap v293: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 3 B/s, 1 objects/s recovering 2024-09-16T09:03:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi111-76450-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi111-76450-48"}]': finished 2024-09-16T09:03:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi111-76430-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi111-76430-39"}]': finished 2024-09-16T09:03:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:40 smithi114 ceph-mon[33315]: osdmap e230: 8 total, 8 up, 8 in 2024-09-16T09:03:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:40 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[29803]: pgmap v293: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 3 B/s, 1 objects/s recovering 2024-09-16T09:03:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi111-76450-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi111-76450-48"}]': finished 2024-09-16T09:03:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi111-76430-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi111-76430-39"}]': finished 2024-09-16T09:03:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[29803]: osdmap e230: 8 total, 8 up, 8 in 2024-09-16T09:03:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[40056]: pgmap v293: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 3 B/s, 1 objects/s recovering 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi111-76450-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi111-76450-48"}]': finished 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi111-76430-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi111-76430-39"}]': finished 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[40056]: osdmap e230: 8 total, 8 up, 8 in 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:40 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:41.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:41 smithi114 ceph-mon[33315]: osdmap e231: 8 total, 8 up, 8 in 2024-09-16T09:03:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:41 smithi111 ceph-mon[29803]: osdmap e231: 8 total, 8 up, 8 in 2024-09-16T09:03:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:41.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:41.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:41.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:41 smithi111 ceph-mon[40056]: osdmap e231: 8 total, 8 up, 8 in 2024-09-16T09:03:41.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:42.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:03:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:03:42.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:42 smithi114 ceph-mon[33315]: pgmap v296: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-31"}]': finished 2024-09-16T09:03:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-31", "mode": "writeback"} : dispatch 2024-09-16T09:03:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:42 smithi114 ceph-mon[33315]: osdmap e232: 8 total, 8 up, 8 in 2024-09-16T09:03:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-31", "mode": "writeback"} : dispatch 2024-09-16T09:03:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[29803]: pgmap v296: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-31"}]': finished 2024-09-16T09:03:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-31", "mode": "writeback"} : dispatch 2024-09-16T09:03:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[29803]: osdmap e232: 8 total, 8 up, 8 in 2024-09-16T09:03:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-31", "mode": "writeback"} : dispatch 2024-09-16T09:03:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[40056]: pgmap v296: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-31"}]': finished 2024-09-16T09:03:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-31", "mode": "writeback"} : dispatch 2024-09-16T09:03:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[40056]: osdmap e232: 8 total, 8 up, 8 in 2024-09-16T09:03:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-31", "mode": "writeback"} : dispatch 2024-09-16T09:03:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-31", "mode": "writeback"}]': finished 2024-09-16T09:03:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi111-76450-48"}]': finished 2024-09-16T09:03:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"}]': finished 2024-09-16T09:03:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: osdmap e233: 8 total, 8 up, 8 in 2024-09-16T09:03:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi111-76450-48"}]': finished 2024-09-16T09:03:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"}]': finished 2024-09-16T09:03:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: osdmap e234: 8 total, 8 up, 8 in 2024-09-16T09:03:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:43.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:03:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:03:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-31", "mode": "writeback"}]': finished 2024-09-16T09:03:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi111-76450-48"}]': finished 2024-09-16T09:03:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"}]': finished 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: osdmap e233: 8 total, 8 up, 8 in 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi111-76450-48"}]': finished 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"}]': finished 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: osdmap e234: 8 total, 8 up, 8 in 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-31", "mode": "writeback"}]': finished 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi111-76450-48"}]': finished 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi111-76430-39"}]': finished 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3738931638' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: osdmap e233: 8 total, 8 up, 8 in 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi111-76450-48"} : dispatch 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"} : dispatch 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:43.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:43.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1992531442' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi111-76450-48"}]': finished 2024-09-16T09:03:43.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi111-76430-39"}]': finished 2024-09-16T09:03:43.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:43.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: osdmap e234: 8 total, 8 up, 8 in 2024-09-16T09:03:43.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:43.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31"} : dispatch 2024-09-16T09:03:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: pgmap v299: 228 pgs: 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:03:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi111-76450-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi111-76450-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi111-76430-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi111-76430-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31"}]': finished 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi111-76450-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi111-76430-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:44.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: osdmap e235: 8 total, 8 up, 8 in 2024-09-16T09:03:44.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "StatRemove_smithi111-76430-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "StatRemove_smithi111-76430-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP_smithi111-76450-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP_smithi111-76450-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: pgmap v299: 228 pgs: 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:03:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi111-76450-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi111-76450-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi111-76430-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi111-76430-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31"}]': finished 2024-09-16T09:03:44.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi111-76450-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi111-76430-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: osdmap e235: 8 total, 8 up, 8 in 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "StatRemove_smithi111-76430-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: pgmap v299: 228 pgs: 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi111-76450-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi111-76450-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi111-76430-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi111-76430-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-31"}]': finished 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi111-76450-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi111-76430-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: osdmap e235: 8 total, 8 up, 8 in 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "StatRemove_smithi111-76430-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "StatRemove_smithi111-76430-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.680 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP_smithi111-76450-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.680 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP_smithi111-76450-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "StatRemove_smithi111-76430-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:44.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP_smithi111-76450-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:44.680 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP_smithi111-76450-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:03:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:03:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:03:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:03:45.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:03:45.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:03:45.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:03:45.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:03:45.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:03:45.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:03:45.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:03:45.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:03:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:46 smithi114 ceph-mon[33315]: pgmap v302: 228 pgs: 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:03:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:46 smithi114 ceph-mon[33315]: osdmap e236: 8 total, 8 up, 8 in 2024-09-16T09:03:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:46 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi111-76430-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi111-76430-40"}]': finished 2024-09-16T09:03:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:46 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi111-76450-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi111-76450-49"}]': finished 2024-09-16T09:03:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:46 smithi114 ceph-mon[33315]: osdmap e237: 8 total, 8 up, 8 in 2024-09-16T09:03:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:46 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[29803]: pgmap v302: 228 pgs: 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:03:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[29803]: osdmap e236: 8 total, 8 up, 8 in 2024-09-16T09:03:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi111-76430-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi111-76430-40"}]': finished 2024-09-16T09:03:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi111-76450-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi111-76450-49"}]': finished 2024-09-16T09:03:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[29803]: osdmap e237: 8 total, 8 up, 8 in 2024-09-16T09:03:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[40056]: pgmap v302: 228 pgs: 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:03:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[40056]: osdmap e236: 8 total, 8 up, 8 in 2024-09-16T09:03:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:03:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi111-76430-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi111-76430-40"}]': finished 2024-09-16T09:03:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi111-76450-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi111-76450-49"}]': finished 2024-09-16T09:03:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[40056]: osdmap e237: 8 total, 8 up, 8 in 2024-09-16T09:03:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:46 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:48.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:48 smithi114 ceph-mon[33315]: pgmap v305: 244 pgs: 48 creating+peering, 196 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:03:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:48 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:48 smithi114 ceph-mon[33315]: osdmap e238: 8 total, 8 up, 8 in 2024-09-16T09:03:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:48 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:48 smithi114 ceph-mon[33315]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[29803]: pgmap v305: 244 pgs: 48 creating+peering, 196 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:03:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[29803]: osdmap e238: 8 total, 8 up, 8 in 2024-09-16T09:03:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[29803]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:48.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[40056]: pgmap v305: 244 pgs: 48 creating+peering, 196 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:03:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[40056]: osdmap e238: 8 total, 8 up, 8 in 2024-09-16T09:03:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:48 smithi111 ceph-mon[40056]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:49 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:49 smithi114 ceph-mon[33315]: osdmap e239: 8 total, 8 up, 8 in 2024-09-16T09:03:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:49 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:49 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:49.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:49 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[29803]: osdmap e239: 8 total, 8 up, 8 in 2024-09-16T09:03:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:49.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:49.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[40056]: osdmap e239: 8 total, 8 up, 8 in 2024-09-16T09:03:49.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:49.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:49.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:49.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:49.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:49.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:49 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:50.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: pgmap v308: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"}]': finished 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-33"}]': finished 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"}]': finished 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-33", "mode": "writeback"} : dispatch 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: osdmap e240: 8 total, 8 up, 8 in 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-33", "mode": "writeback"} : dispatch 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: pgmap v308: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"}]': finished 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-33"}]': finished 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"}]': finished 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-33", "mode": "writeback"} : dispatch 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: osdmap e240: 8 total, 8 up, 8 in 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-33", "mode": "writeback"} : dispatch 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: pgmap v308: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi111-76450-49"}]': finished 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-33"}]': finished 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi111-76430-40"}]': finished 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3298441453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-33", "mode": "writeback"} : dispatch 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: osdmap e240: 8 total, 8 up, 8 in 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"} : dispatch 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-33", "mode": "writeback"} : dispatch 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4208426776' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"} : dispatch 2024-09-16T09:03:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"}]': finished 2024-09-16T09:03:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-33", "mode": "writeback"}]': finished 2024-09-16T09:03:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"}]': finished 2024-09-16T09:03:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: osdmap e241: 8 total, 8 up, 8 in 2024-09-16T09:03:51.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi111-76450-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi111-76430-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi111-76450-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:51.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"}]': finished 2024-09-16T09:03:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-33", "mode": "writeback"}]': finished 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"}]': finished 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: osdmap e241: 8 total, 8 up, 8 in 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi111-76450-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi111-76430-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi111-76450-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi111-76430-40"}]': finished 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-33", "mode": "writeback"}]': finished 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi111-76450-49"}]': finished 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: osdmap e241: 8 total, 8 up, 8 in 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi111-76450-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi111-76430-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:51.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi111-76450-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:51.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:51.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:03:51.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:03:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:03:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: pgmap v311: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi111-76430-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi111-76450-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP2_smithi111-76450-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: osdmap e242: 8 total, 8 up, 8 in 2024-09-16T09:03:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ExecuteClass_smithi111-76430-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:52.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP2_smithi111-76450-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: pgmap v311: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi111-76430-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi111-76450-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP2_smithi111-76450-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: osdmap e242: 8 total, 8 up, 8 in 2024-09-16T09:03:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ExecuteClass_smithi111-76430-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP2_smithi111-76450-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: pgmap v311: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi111-76430-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi111-76450-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP2_smithi111-76450-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: osdmap e242: 8 total, 8 up, 8 in 2024-09-16T09:03:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ExecuteClass_smithi111-76430-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:52.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33"} : dispatch 2024-09-16T09:03:52.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripPP2_smithi111-76450-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:53.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:53 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33"}]': finished 2024-09-16T09:03:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:53 smithi114 ceph-mon[33315]: osdmap e243: 8 total, 8 up, 8 in 2024-09-16T09:03:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:53 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi111-76430-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi111-76430-41"}]': finished 2024-09-16T09:03:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi111-76450-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi111-76450-50"}]': finished 2024-09-16T09:03:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:53 smithi114 ceph-mon[33315]: osdmap e244: 8 total, 8 up, 8 in 2024-09-16T09:03:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:03:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:03:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33"}]': finished 2024-09-16T09:03:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[29803]: osdmap e243: 8 total, 8 up, 8 in 2024-09-16T09:03:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi111-76430-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi111-76430-41"}]': finished 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi111-76450-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi111-76450-50"}]': finished 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[29803]: osdmap e244: 8 total, 8 up, 8 in 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-33"}]': finished 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[40056]: osdmap e243: 8 total, 8 up, 8 in 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi111-76430-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi111-76430-41"}]': finished 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi111-76450-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi111-76450-50"}]': finished 2024-09-16T09:03:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:53 smithi111 ceph-mon[40056]: osdmap e244: 8 total, 8 up, 8 in 2024-09-16T09:03:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:54 smithi114 ceph-mon[33315]: pgmap v314: 228 pgs: 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-16T09:03:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:54 smithi114 ceph-mon[33315]: osdmap e245: 8 total, 8 up, 8 in 2024-09-16T09:03:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:54 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:54 smithi111 ceph-mon[29803]: pgmap v314: 228 pgs: 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-16T09:03:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:54 smithi111 ceph-mon[29803]: osdmap e245: 8 total, 8 up, 8 in 2024-09-16T09:03:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:54 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:54.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:54 smithi111 ceph-mon[40056]: pgmap v314: 228 pgs: 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-16T09:03:54.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:54 smithi111 ceph-mon[40056]: osdmap e245: 8 total, 8 up, 8 in 2024-09-16T09:03:54.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:54.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:54 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:56 smithi114 ceph-mon[33315]: pgmap v317: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:03:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:56 smithi114 ceph-mon[33315]: osdmap e246: 8 total, 8 up, 8 in 2024-09-16T09:03:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:56.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[29803]: pgmap v317: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:03:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[29803]: osdmap e246: 8 total, 8 up, 8 in 2024-09-16T09:03:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[40056]: pgmap v317: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:03:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:03:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[40056]: osdmap e246: 8 total, 8 up, 8 in 2024-09-16T09:03:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:03:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:57.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi111-76430-41"}]': finished 2024-09-16T09:03:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"}]': finished 2024-09-16T09:03:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:57 smithi114 ceph-mon[33315]: osdmap e247: 8 total, 8 up, 8 in 2024-09-16T09:03:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:03:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:03:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi111-76430-41"}]': finished 2024-09-16T09:03:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"}]': finished 2024-09-16T09:03:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[29803]: osdmap e247: 8 total, 8 up, 8 in 2024-09-16T09:03:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:03:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi111-76430-41"}]': finished 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi111-76450-50"}]': finished 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[40056]: osdmap e247: 8 total, 8 up, 8 in 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3690390100' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi111-76430-41"} : dispatch 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:03:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"} : dispatch 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: pgmap v320: 228 pgs: 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi111-76430-41"}]': finished 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"}]': finished 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"}]': finished 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-35", "mode": "writeback"} : dispatch 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: osdmap e248: 8 total, 8 up, 8 in 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-35", "mode": "writeback"} : dispatch 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi111-76430-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-35", "mode": "writeback"}]': finished 2024-09-16T09:03:58.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi111-76430-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:58.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: osdmap e249: 8 total, 8 up, 8 in 2024-09-16T09:03:58.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "MultiWrite_smithi111-76430-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:03:58.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/55604784' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:58.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:03:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: pgmap v320: 228 pgs: 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:03:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi111-76430-41"}]': finished 2024-09-16T09:03:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"}]': finished 2024-09-16T09:03:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"}]': finished 2024-09-16T09:03:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-35", "mode": "writeback"} : dispatch 2024-09-16T09:03:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: osdmap e248: 8 total, 8 up, 8 in 2024-09-16T09:03:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-35", "mode": "writeback"} : dispatch 2024-09-16T09:03:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi111-76430-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-35", "mode": "writeback"}]': finished 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi111-76430-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: osdmap e249: 8 total, 8 up, 8 in 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "MultiWrite_smithi111-76430-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/55604784' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: pgmap v320: 228 pgs: 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:03:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/562223453' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi111-76430-41"}]': finished 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"}]': finished 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi111-76450-50"}]': finished 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-35", "mode": "writeback"} : dispatch 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: osdmap e248: 8 total, 8 up, 8 in 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-35", "mode": "writeback"} : dispatch 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi111-76430-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-35", "mode": "writeback"}]': finished 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi111-76430-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: osdmap e249: 8 total, 8 up, 8 in 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "MultiWrite_smithi111-76430-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/55604784' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:03:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:03:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:00.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:00 smithi114 ceph-mon[33315]: pgmap v323: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:00 smithi114 ceph-mon[33315]: osdmap e250: 8 total, 8 up, 8 in 2024-09-16T09:04:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:00 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:00 smithi111 ceph-mon[29803]: pgmap v323: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:00 smithi111 ceph-mon[29803]: osdmap e250: 8 total, 8 up, 8 in 2024-09-16T09:04:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:00 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:00.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:00 smithi111 ceph-mon[40056]: pgmap v323: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:00.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi111-76450-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:00.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:00 smithi111 ceph-mon[40056]: osdmap e250: 8 total, 8 up, 8 in 2024-09-16T09:04:00.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:00 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:01.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi111-76430-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi111-76430-42"}]': finished 2024-09-16T09:04:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-mon[33315]: osdmap e251: 8 total, 8 up, 8 in 2024-09-16T09:04:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi111-76430-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi111-76430-42"}]': finished 2024-09-16T09:04:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[29803]: osdmap e251: 8 total, 8 up, 8 in 2024-09-16T09:04:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi111-76430-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi111-76430-42"}]': finished 2024-09-16T09:04:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[40056]: osdmap e251: 8 total, 8 up, 8 in 2024-09-16T09:04:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:02.163 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:04:01 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:04:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:02 smithi114 ceph-mon[33315]: pgmap v326: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:02 smithi114 ceph-mon[33315]: osdmap e252: 8 total, 8 up, 8 in 2024-09-16T09:04:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi111-76450-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:02.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:02.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[29803]: pgmap v326: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[29803]: osdmap e252: 8 total, 8 up, 8 in 2024-09-16T09:04:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi111-76450-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[40056]: pgmap v326: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[40056]: osdmap e252: 8 total, 8 up, 8 in 2024-09-16T09:04:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi111-76450-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"}]': finished 2024-09-16T09:04:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:03 smithi114 ceph-mon[33315]: osdmap e253: 8 total, 8 up, 8 in 2024-09-16T09:04:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:04:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:03 smithi114 ceph-mon[33315]: pgmap v329: 228 pgs: 228 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-16T09:04:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:04:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:04:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:04:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"}]': finished 2024-09-16T09:04:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:03 smithi111 ceph-mon[29803]: osdmap e253: 8 total, 8 up, 8 in 2024-09-16T09:04:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:04:03.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:03 smithi111 ceph-mon[29803]: pgmap v329: 228 pgs: 228 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-16T09:04:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-35"}]': finished 2024-09-16T09:04:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:03 smithi111 ceph-mon[40056]: osdmap e253: 8 total, 8 up, 8 in 2024-09-16T09:04:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:04:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:03 smithi111 ceph-mon[40056]: pgmap v329: 228 pgs: 228 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6981 ms) 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7031 ms) 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6993 ms) 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7001 ms) 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7039 ms) 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7026 ms) 2024-09-16T09:04:04.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7027 ms) 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6907 ms) 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7006 ms) 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7039 ms) 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6953 ms) 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7012 ms) 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7010 ms) 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-16T09:04:04.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7024 ms) 2024-09-16T09:04:04.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-16T09:04:04.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7032 ms) 2024-09-16T09:04:04.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-16T09:04:04.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7031 ms) 2024-09-16T09:04:04.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112114 ms total) 2024-09-16T09:04:04.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: 2024-09-16T09:04:04.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [----------] Global test environment tear-down 2024-09-16T09:04:04.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (189515 ms total) 2024-09-16T09:04:04.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-16T09:04:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi111-76450-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi111-76450-52"}]': finished 2024-09-16T09:04:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi111-76430-42"}]': finished 2024-09-16T09:04:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:04 smithi114 ceph-mon[33315]: osdmap e254: 8 total, 8 up, 8 in 2024-09-16T09:04:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:04:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi111-76450-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi111-76450-52"}]': finished 2024-09-16T09:04:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi111-76430-42"}]': finished 2024-09-16T09:04:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[29803]: osdmap e254: 8 total, 8 up, 8 in 2024-09-16T09:04:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:04:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:04.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi111-76450-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi111-76450-52"}]': finished 2024-09-16T09:04:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi111-76430-42"}]': finished 2024-09-16T09:04:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[40056]: osdmap e254: 8 total, 8 up, 8 in 2024-09-16T09:04:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWrite_smithi111-76430-42"} : dispatch 2024-09-16T09:04:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi111-76430-42"}]': finished 2024-09-16T09:04:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:05 smithi114 ceph-mon[33315]: osdmap e255: 8 total, 8 up, 8 in 2024-09-16T09:04:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:05 smithi114 ceph-mon[33315]: pgmap v332: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-16T09:04:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi111-76430-42"}]': finished 2024-09-16T09:04:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[29803]: osdmap e255: 8 total, 8 up, 8 in 2024-09-16T09:04:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:05.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[29803]: pgmap v332: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-16T09:04:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4202582395' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi111-76430-42"}]': finished 2024-09-16T09:04:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[40056]: osdmap e255: 8 total, 8 up, 8 in 2024-09-16T09:04:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35"} : dispatch 2024-09-16T09:04:05.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:05 smithi111 ceph-mon[40056]: pgmap v332: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-16T09:04:06.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:06 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:06 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:04:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35"}]': finished 2024-09-16T09:04:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:06 smithi114 ceph-mon[33315]: osdmap e256: 8 total, 8 up, 8 in 2024-09-16T09:04:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:04:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35"}]': finished 2024-09-16T09:04:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[29803]: osdmap e256: 8 total, 8 up, 8 in 2024-09-16T09:04:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:04:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-35"}]': finished 2024-09-16T09:04:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[40056]: osdmap e256: 8 total, 8 up, 8 in 2024-09-16T09:04:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:07.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52"}]': finished 2024-09-16T09:04:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:07 smithi114 ceph-mon[33315]: osdmap e257: 8 total, 8 up, 8 in 2024-09-16T09:04:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:07 smithi114 ceph-mon[33315]: pgmap v335: 196 pgs: 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:04:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52"}]': finished 2024-09-16T09:04:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:07 smithi111 ceph-mon[29803]: osdmap e257: 8 total, 8 up, 8 in 2024-09-16T09:04:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:07 smithi111 ceph-mon[29803]: pgmap v335: 196 pgs: 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:04:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi111-76450-52"}]': finished 2024-09-16T09:04:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:07 smithi111 ceph-mon[40056]: osdmap e257: 8 total, 8 up, 8 in 2024-09-16T09:04:07.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi111-76450-52"} : dispatch 2024-09-16T09:04:07.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:07 smithi111 ceph-mon[40056]: pgmap v335: 196 pgs: 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi111-76450-52"}]': finished 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: osdmap e258: 8 total, 8 up, 8 in 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: osdmap e259: 8 total, 8 up, 8 in 2024-09-16T09:04:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi111-76450-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi111-76450-52"}]': finished 2024-09-16T09:04:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: osdmap e258: 8 total, 8 up, 8 in 2024-09-16T09:04:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: osdmap e259: 8 total, 8 up, 8 in 2024-09-16T09:04:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi111-76450-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1875817807' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi111-76450-52"}]': finished 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: osdmap e258: 8 total, 8 up, 8 in 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: osdmap e259: 8 total, 8 up, 8 in 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi111-76450-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:08.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:10 smithi114 ceph-mon[33315]: pgmap v338: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:04:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:10 smithi114 ceph-mon[33315]: osdmap e260: 8 total, 8 up, 8 in 2024-09-16T09:04:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[29803]: pgmap v338: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:04:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[29803]: osdmap e260: 8 total, 8 up, 8 in 2024-09-16T09:04:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[40056]: pgmap v338: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:04:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[40056]: osdmap e260: 8 total, 8 up, 8 in 2024-09-16T09:04:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi111-76450-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi111-76450-53"}]': finished 2024-09-16T09:04:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-37"}]': finished 2024-09-16T09:04:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:11 smithi114 ceph-mon[33315]: osdmap e261: 8 total, 8 up, 8 in 2024-09-16T09:04:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-37", "mode": "writeback"} : dispatch 2024-09-16T09:04:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-37", "mode": "writeback"} : dispatch 2024-09-16T09:04:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:11 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:04:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-37", "mode": "writeback"}]': finished 2024-09-16T09:04:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi111-76450-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi111-76450-53"}]': finished 2024-09-16T09:04:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-37"}]': finished 2024-09-16T09:04:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[29803]: osdmap e261: 8 total, 8 up, 8 in 2024-09-16T09:04:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-37", "mode": "writeback"} : dispatch 2024-09-16T09:04:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-37", "mode": "writeback"} : dispatch 2024-09-16T09:04:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:04:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-37", "mode": "writeback"}]': finished 2024-09-16T09:04:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi111-76450-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi111-76450-53"}]': finished 2024-09-16T09:04:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-37"}]': finished 2024-09-16T09:04:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[40056]: osdmap e261: 8 total, 8 up, 8 in 2024-09-16T09:04:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-37", "mode": "writeback"} : dispatch 2024-09-16T09:04:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-37", "mode": "writeback"} : dispatch 2024-09-16T09:04:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:04:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-37", "mode": "writeback"}]': finished 2024-09-16T09:04:12.163 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:04:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:04:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:12 smithi114 ceph-mon[33315]: pgmap v341: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:12 smithi114 ceph-mon[33315]: osdmap e262: 8 total, 8 up, 8 in 2024-09-16T09:04:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:12 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:12 smithi114 ceph-mon[33315]: osdmap e263: 8 total, 8 up, 8 in 2024-09-16T09:04:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[29803]: pgmap v341: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[29803]: osdmap e262: 8 total, 8 up, 8 in 2024-09-16T09:04:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[29803]: osdmap e263: 8 total, 8 up, 8 in 2024-09-16T09:04:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[40056]: pgmap v341: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[40056]: osdmap e262: 8 total, 8 up, 8 in 2024-09-16T09:04:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:12.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:12.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[40056]: osdmap e263: 8 total, 8 up, 8 in 2024-09-16T09:04:12.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37"} : dispatch 2024-09-16T09:04:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:13 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:04:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37"}]': finished 2024-09-16T09:04:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53"}]': finished 2024-09-16T09:04:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:13 smithi114 ceph-mon[33315]: osdmap e264: 8 total, 8 up, 8 in 2024-09-16T09:04:13.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:13.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:04:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:04:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:04:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37"}]': finished 2024-09-16T09:04:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53"}]': finished 2024-09-16T09:04:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[29803]: osdmap e264: 8 total, 8 up, 8 in 2024-09-16T09:04:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:04:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-37"}]': finished 2024-09-16T09:04:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi111-76450-53"}]': finished 2024-09-16T09:04:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[40056]: osdmap e264: 8 total, 8 up, 8 in 2024-09-16T09:04:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi111-76450-53"} : dispatch 2024-09-16T09:04:14.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:14 smithi114 ceph-mon[33315]: pgmap v344: 228 pgs: 228 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-09-16T09:04:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi111-76450-53"}]': finished 2024-09-16T09:04:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:14 smithi114 ceph-mon[33315]: osdmap e265: 8 total, 8 up, 8 in 2024-09-16T09:04:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:14 smithi111 ceph-mon[29803]: pgmap v344: 228 pgs: 228 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-09-16T09:04:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi111-76450-53"}]': finished 2024-09-16T09:04:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:14 smithi111 ceph-mon[29803]: osdmap e265: 8 total, 8 up, 8 in 2024-09-16T09:04:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:14 smithi111 ceph-mon[40056]: pgmap v344: 228 pgs: 228 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-09-16T09:04:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3287520361' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi111-76450-53"}]': finished 2024-09-16T09:04:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:14 smithi111 ceph-mon[40056]: osdmap e265: 8 total, 8 up, 8 in 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi111-76450-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi111-76450-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi111-76450-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: osdmap e266: 8 total, 8 up, 8 in 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsCompletePP_smithi111-76450-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsCompletePP_smithi111-76450-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi111-76450-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi111-76450-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi111-76450-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: osdmap e266: 8 total, 8 up, 8 in 2024-09-16T09:04:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsCompletePP_smithi111-76450-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsCompletePP_smithi111-76450-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi111-76450-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi111-76450-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi111-76450-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:15.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: osdmap e266: 8 total, 8 up, 8 in 2024-09-16T09:04:15.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsCompletePP_smithi111-76450-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:15.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsCompletePP_smithi111-76450-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:15.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:16.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:16 smithi114 ceph-mon[33315]: pgmap v347: 196 pgs: 196 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:04:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:16 smithi114 ceph-mon[33315]: osdmap e267: 8 total, 8 up, 8 in 2024-09-16T09:04:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[29803]: pgmap v347: 196 pgs: 196 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:04:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[29803]: osdmap e267: 8 total, 8 up, 8 in 2024-09-16T09:04:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[40056]: pgmap v347: 196 pgs: 196 active+clean; 4.6 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:04:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[40056]: osdmap e267: 8 total, 8 up, 8 in 2024-09-16T09:04:16.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:18 smithi114 ceph-mon[33315]: pgmap v350: 228 pgs: 228 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi111-76450-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi111-76450-54"}]': finished 2024-09-16T09:04:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:18 smithi114 ceph-mon[33315]: osdmap e268: 8 total, 8 up, 8 in 2024-09-16T09:04:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:18 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[29803]: pgmap v350: 228 pgs: 228 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi111-76450-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi111-76450-54"}]': finished 2024-09-16T09:04:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[29803]: osdmap e268: 8 total, 8 up, 8 in 2024-09-16T09:04:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:18.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[40056]: pgmap v350: 228 pgs: 228 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:18.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi111-76450-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi111-76450-54"}]': finished 2024-09-16T09:04:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[40056]: osdmap e268: 8 total, 8 up, 8 in 2024-09-16T09:04:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:18 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:19.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-39"}]': finished 2024-09-16T09:04:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-39", "mode": "writeback"} : dispatch 2024-09-16T09:04:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:19 smithi114 ceph-mon[33315]: osdmap e269: 8 total, 8 up, 8 in 2024-09-16T09:04:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-39", "mode": "writeback"} : dispatch 2024-09-16T09:04:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:19 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:04:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-39"}]': finished 2024-09-16T09:04:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-39", "mode": "writeback"} : dispatch 2024-09-16T09:04:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[40056]: osdmap e269: 8 total, 8 up, 8 in 2024-09-16T09:04:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-39", "mode": "writeback"} : dispatch 2024-09-16T09:04:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:04:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-39"}]': finished 2024-09-16T09:04:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-39", "mode": "writeback"} : dispatch 2024-09-16T09:04:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[29803]: osdmap e269: 8 total, 8 up, 8 in 2024-09-16T09:04:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-39", "mode": "writeback"} : dispatch 2024-09-16T09:04:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:19 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:04:20.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:20 smithi114 ceph-mon[33315]: pgmap v353: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-39", "mode": "writeback"}]': finished 2024-09-16T09:04:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:20 smithi114 ceph-mon[33315]: osdmap e270: 8 total, 8 up, 8 in 2024-09-16T09:04:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[40056]: pgmap v353: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-39", "mode": "writeback"}]': finished 2024-09-16T09:04:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[40056]: osdmap e270: 8 total, 8 up, 8 in 2024-09-16T09:04:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[29803]: pgmap v353: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-39", "mode": "writeback"}]': finished 2024-09-16T09:04:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[29803]: osdmap e270: 8 total, 8 up, 8 in 2024-09-16T09:04:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"}]': finished 2024-09-16T09:04:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:21 smithi114 ceph-mon[33315]: osdmap e271: 8 total, 8 up, 8 in 2024-09-16T09:04:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:21.565 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:21.566 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"}]': finished 2024-09-16T09:04:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[29803]: osdmap e271: 8 total, 8 up, 8 in 2024-09-16T09:04:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi111-76450-54"}]': finished 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[40056]: osdmap e271: 8 total, 8 up, 8 in 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3818747489' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"} : dispatch 2024-09-16T09:04:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39"} : dispatch 2024-09-16T09:04:21.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:04:21 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: seed 76716 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (364 ms) 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (9978 ms) 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (80 ms) 2024-09-16T09:04:22.319 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10422 ms total) 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7037 ms) 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8010 ms) 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10002 ms) 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: my_snaps [3] 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: my_snaps [4,3] 2024-09-16T09:04:22.320 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: my_snaps [5,4,3] 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: promoting some heads 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: promoting from clones for snap 3 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: done waiting 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (45981 ms) 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10001 ms) 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-16T09:04:22.321 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7990 ms) 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8055 ms) 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8026 ms) 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10066 ms) 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8873 ms) 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10056 ms) 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12954 ms) 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-16T09:04:22.322 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8020 ms) 2024-09-16T09:04:22.323 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-16T09:04:22.323 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8036 ms) 2024-09-16T09:04:22.323 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-16T09:04:22.323 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13065 ms) 2024-09-16T09:04:22.323 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-16T09:04:22.323 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7890 ms) 2024-09-16T09:04:22.323 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-16T09:04:22.323 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8046 ms) 2024-09-16T09:04:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: pgmap v356: 228 pgs: 228 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:04:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"}]': finished 2024-09-16T09:04:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39"}]': finished 2024-09-16T09:04:22.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: osdmap e272: 8 total, 8 up, 8 in 2024-09-16T09:04:22.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi111-76450-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:22.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi111-76450-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: pgmap v356: 228 pgs: 228 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:04:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"}]': finished 2024-09-16T09:04:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39"}]': finished 2024-09-16T09:04:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: osdmap e272: 8 total, 8 up, 8 in 2024-09-16T09:04:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi111-76450-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi111-76450-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: pgmap v356: 228 pgs: 228 active+clean; 4.6 MiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi111-76450-54"}]': finished 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-39"}]': finished 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: osdmap e272: 8 total, 8 up, 8 in 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi111-76450-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:22.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi111-76450-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:23.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:23 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi111-76450-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:23 smithi114 ceph-mon[33315]: osdmap e273: 8 total, 8 up, 8 in 2024-09-16T09:04:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsSafePP_smithi111-76450-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsSafePP_smithi111-76450-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:23.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:04:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:04:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi111-76450-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[29803]: osdmap e273: 8 total, 8 up, 8 in 2024-09-16T09:04:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsSafePP_smithi111-76450-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsSafePP_smithi111-76450-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi111-76450-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[40056]: osdmap e273: 8 total, 8 up, 8 in 2024-09-16T09:04:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsSafePP_smithi111-76450-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "IsSafePP_smithi111-76450-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:24 smithi114 ceph-mon[33315]: pgmap v359: 196 pgs: 196 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-09-16T09:04:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:24 smithi114 ceph-mon[33315]: osdmap e274: 8 total, 8 up, 8 in 2024-09-16T09:04:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:24 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:24 smithi111 ceph-mon[29803]: pgmap v359: 196 pgs: 196 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-09-16T09:04:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:24 smithi111 ceph-mon[29803]: osdmap e274: 8 total, 8 up, 8 in 2024-09-16T09:04:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:24 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:24 smithi111 ceph-mon[40056]: pgmap v359: 196 pgs: 196 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-09-16T09:04:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:24.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:24 smithi111 ceph-mon[40056]: osdmap e274: 8 total, 8 up, 8 in 2024-09-16T09:04:24.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:24 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi111-76450-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi111-76450-55"}]': finished 2024-09-16T09:04:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:25 smithi114 ceph-mon[33315]: osdmap e275: 8 total, 8 up, 8 in 2024-09-16T09:04:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:25 smithi114 ceph-mon[33315]: osdmap e276: 8 total, 8 up, 8 in 2024-09-16T09:04:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:04:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:04:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi111-76450-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi111-76450-55"}]': finished 2024-09-16T09:04:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[29803]: osdmap e275: 8 total, 8 up, 8 in 2024-09-16T09:04:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[29803]: osdmap e276: 8 total, 8 up, 8 in 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi111-76450-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi111-76450-55"}]': finished 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[40056]: osdmap e275: 8 total, 8 up, 8 in 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[40056]: osdmap e276: 8 total, 8 up, 8 in 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:04:25.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:04:26.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:26 smithi114 ceph-mon[33315]: pgmap v362: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-09-16T09:04:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:26 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:04:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:26 smithi114 ceph-mon[33315]: osdmap e277: 8 total, 8 up, 8 in 2024-09-16T09:04:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[29803]: pgmap v362: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-09-16T09:04:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:04:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[29803]: osdmap e277: 8 total, 8 up, 8 in 2024-09-16T09:04:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[40056]: pgmap v362: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-09-16T09:04:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:04:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[40056]: osdmap e277: 8 total, 8 up, 8 in 2024-09-16T09:04:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:28 smithi114 ceph-mon[33315]: pgmap v365: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:04:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"}]': finished 2024-09-16T09:04:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:28 smithi114 ceph-mon[33315]: osdmap e278: 8 total, 8 up, 8 in 2024-09-16T09:04:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:04:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:04:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[29803]: pgmap v365: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:04:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"}]': finished 2024-09-16T09:04:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[29803]: osdmap e278: 8 total, 8 up, 8 in 2024-09-16T09:04:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:04:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:04:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[40056]: pgmap v365: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:04:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi111-76450-55"}]': finished 2024-09-16T09:04:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[40056]: osdmap e278: 8 total, 8 up, 8 in 2024-09-16T09:04:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1502144097' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"} : dispatch 2024-09-16T09:04:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:04:28.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:04:29.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"}]': finished 2024-09-16T09:04:29.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:04:29.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:29 smithi114 ceph-mon[33315]: osdmap e279: 8 total, 8 up, 8 in 2024-09-16T09:04:29.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:29.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:29.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi111-76450-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:29.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:29 smithi114 ceph-mon[33315]: pgmap v368: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:29.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"}]': finished 2024-09-16T09:04:29.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:04:29.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[29803]: osdmap e279: 8 total, 8 up, 8 in 2024-09-16T09:04:29.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:29.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:29.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi111-76450-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:29.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[29803]: pgmap v368: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:29.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi111-76450-55"}]': finished 2024-09-16T09:04:29.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:04:29.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[40056]: osdmap e279: 8 total, 8 up, 8 in 2024-09-16T09:04:29.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:29.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:29.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi111-76450-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:29.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:29 smithi111 ceph-mon[40056]: pgmap v368: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi111-76450-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:30 smithi114 ceph-mon[33315]: osdmap e280: 8 total, 8 up, 8 in 2024-09-16T09:04:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReturnValuePP_smithi111-76450-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41"} : dispatch 2024-09-16T09:04:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41"} : dispatch 2024-09-16T09:04:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi111-76450-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[29803]: osdmap e280: 8 total, 8 up, 8 in 2024-09-16T09:04:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReturnValuePP_smithi111-76450-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41"} : dispatch 2024-09-16T09:04:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41"} : dispatch 2024-09-16T09:04:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi111-76450-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[40056]: osdmap e280: 8 total, 8 up, 8 in 2024-09-16T09:04:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReturnValuePP_smithi111-76450-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41"} : dispatch 2024-09-16T09:04:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41"} : dispatch 2024-09-16T09:04:31.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41"}]': finished 2024-09-16T09:04:31.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:31 smithi114 ceph-mon[33315]: osdmap e281: 8 total, 8 up, 8 in 2024-09-16T09:04:31.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:31 smithi114 ceph-mon[33315]: pgmap v371: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:31.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:31.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi111-76450-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi111-76450-56"}]': finished 2024-09-16T09:04:31.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:31 smithi114 ceph-mon[33315]: osdmap e282: 8 total, 8 up, 8 in 2024-09-16T09:04:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41"}]': finished 2024-09-16T09:04:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[29803]: osdmap e281: 8 total, 8 up, 8 in 2024-09-16T09:04:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[29803]: pgmap v371: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi111-76450-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi111-76450-56"}]': finished 2024-09-16T09:04:31.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[29803]: osdmap e282: 8 total, 8 up, 8 in 2024-09-16T09:04:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-41"}]': finished 2024-09-16T09:04:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[40056]: osdmap e281: 8 total, 8 up, 8 in 2024-09-16T09:04:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[40056]: pgmap v371: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi111-76450-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi111-76450-56"}]': finished 2024-09-16T09:04:31.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:31 smithi111 ceph-mon[40056]: osdmap e282: 8 total, 8 up, 8 in 2024-09-16T09:04:32.163 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:04:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:04:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:32 smithi114 ceph-mon[33315]: osdmap e283: 8 total, 8 up, 8 in 2024-09-16T09:04:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:32 smithi111 ceph-mon[29803]: osdmap e283: 8 total, 8 up, 8 in 2024-09-16T09:04:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:32 smithi111 ceph-mon[40056]: osdmap e283: 8 total, 8 up, 8 in 2024-09-16T09:04:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:33.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:33 smithi114 ceph-mon[33315]: pgmap v374: 236 pgs: 32 unknown, 8 creating+peering, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:33 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:33 smithi114 ceph-mon[33315]: osdmap e284: 8 total, 8 up, 8 in 2024-09-16T09:04:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool get","pool":"test-rados-api-smithi111-76716-6","var": "pg_num","format": "json"} : dispatch 2024-09-16T09:04:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:04:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:04:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[29803]: pgmap v374: 236 pgs: 32 unknown, 8 creating+peering, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[29803]: osdmap e284: 8 total, 8 up, 8 in 2024-09-16T09:04:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool get","pool":"test-rados-api-smithi111-76716-6","var": "pg_num","format": "json"} : dispatch 2024-09-16T09:04:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:33.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:33.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[40056]: pgmap v374: 236 pgs: 32 unknown, 8 creating+peering, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:33.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:33.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[40056]: osdmap e284: 8 total, 8 up, 8 in 2024-09-16T09:04:33.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool get","pool":"test-rados-api-smithi111-76716-6","var": "pg_num","format": "json"} : dispatch 2024-09-16T09:04:33.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:33.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:33.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi111-76450-56"}]': finished 2024-09-16T09:04:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_count","val": "8"} : dispatch 2024-09-16T09:04:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:35 smithi114 ceph-mon[33315]: osdmap e285: 8 total, 8 up, 8 in 2024-09-16T09:04:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_count","val": "8"} : dispatch 2024-09-16T09:04:35.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:35 smithi114 ceph-mon[33315]: pgmap v377: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi111-76450-56"}]': finished 2024-09-16T09:04:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_count","val": "8"} : dispatch 2024-09-16T09:04:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[29803]: osdmap e285: 8 total, 8 up, 8 in 2024-09-16T09:04:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_count","val": "8"} : dispatch 2024-09-16T09:04:35.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[29803]: pgmap v377: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi111-76450-56"}]': finished 2024-09-16T09:04:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_count","val": "8"} : dispatch 2024-09-16T09:04:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[40056]: osdmap e285: 8 total, 8 up, 8 in 2024-09-16T09:04:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi111-76450-56"} : dispatch 2024-09-16T09:04:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_count","val": "8"} : dispatch 2024-09-16T09:04:35.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:35 smithi111 ceph-mon[40056]: pgmap v377: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi111-76450-56"}]': finished 2024-09-16T09:04:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_count","val": "8"}]': finished 2024-09-16T09:04:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: osdmap e286: 8 total, 8 up, 8 in 2024-09-16T09:04:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi111-76450-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:36.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi111-76450-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi111-76450-56"}]': finished 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_count","val": "8"}]': finished 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: osdmap e286: 8 total, 8 up, 8 in 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi111-76450-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi111-76450-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1693818387' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi111-76450-56"}]': finished 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_count","val": "8"}]': finished 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: osdmap e286: 8 total, 8 up, 8 in 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi111-76450-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:36.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi111-76450-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:37.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-16T09:04:37.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T09:04:37.600 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: ok, hit_set contains 273:602f83fe:::foo:head 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9034 ms) 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg_num = 32 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 0 ls 1726477478,0 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 1 ls 1726477478,0 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 2 ls 1726477478,0 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 3 ls 1726477478,0 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 4 ls 1726477478,0 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 5 ls 1726477478,0 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 6 ls 1726477478,0 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 7 ls 1726477478,0 2024-09-16T09:04:37.601 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 8 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 9 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 10 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 11 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 12 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 13 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 14 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 15 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 16 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 17 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 18 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 19 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 20 ls 1726477478,0 2024-09-16T09:04:37.602 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 21 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 22 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 23 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 24 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 25 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 26 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 27 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 28 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 29 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 30 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg 31 ls 1726477478,0 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: pg_num = 32 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:6cac518f:::0:head 2024-09-16T09:04:37.603 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:02547ec2:::1:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:f905c69b:::2:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:cfc208b3:::3:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d83876eb:::4:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:b29083e3:::5:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c4fdafeb:::6:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:5c6b0b28:::7:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:bd63b0f1:::8:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:e960b815:::9:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:52ea6a34:::10:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:89d3ae78:::11:head 2024-09-16T09:04:37.604 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:de5d7c5f:::12:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:566253c9:::13:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:62a1935d:::14:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:863748b0:::15:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:3958e169:::16:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:4d4dabf9:::17:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:8391935d:::18:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:28883081:::19:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:69259c59:::20:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:4bdb80b7:::21:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:a11c5d71:::22:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:271af37b:::23:head 2024-09-16T09:04:37.605 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:95b121be:::24:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:58d1031b:::25:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:0a050783:::26:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c709704c:::27:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:cbe56eaf:::28:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:86b4b162:::29:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:70d89383:::30:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:dd450c7c:::31:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:6d5729b1:::32:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c388f3fb:::33:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:56cfea31:::34:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:9dbc1bf7:::35:head 2024-09-16T09:04:37.606 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:40b74ccd:::36:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:4d5aaf42:::37:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:920f362c:::38:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:6cc53222:::39:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:9cad833f:::40:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:1ea84d41:::41:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c4480ef6:::42:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:a694361e:::43:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d1bd33e9:::44:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:ddc2cd5d:::45:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:2b782207:::46:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:7b187fca:::47:head 2024-09-16T09:04:37.607 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:90ecdf6f:::48:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:a5ed95fe:::49:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:ea0eaa55:::50:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:f33ef17b:::51:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:a0d1b2f6:::52:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:60c5229e:::53:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:edcbc575:::54:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:102cf253:::55:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:efb7fb0b:::56:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:50d0a326:::57:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d4dc5daf:::58:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:3a130462:::59:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:ec87ed71:::60:head 2024-09-16T09:04:37.608 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d5bc9454:::61:head 2024-09-16T09:04:37.609 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:3ddfe313:::62:head 2024-09-16T09:04:37.609 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:7c2816b9:::63:head 2024-09-16T09:04:37.609 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:47e00e4d:::64:head 2024-09-16T09:04:37.609 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c6410c18:::65:head 2024-09-16T09:04:37.609 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:b48ed237:::66:head 2024-09-16T09:04:37.609 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:cd63ad31:::67:head 2024-09-16T09:04:37.609 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:b179e92b:::68:head 2024-09-16T09:04:37.612 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:0d9f741a:::69:head 2024-09-16T09:04:37.612 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:6d3352ae:::70:head 2024-09-16T09:04:37.612 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c6d5c19e:::71:head 2024-09-16T09:04:37.612 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:bc4729c3:::72:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:77e930b9:::73:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:0abeecfd:::74:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:b7c37e15:::75:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:b6378398:::76:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:02bd68de:::77:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:cc795d2d:::78:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:630d4fea:::79:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:e0d29ef5:::80:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:fd6f13d2:::81:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:606461d5:::82:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:eadbdc43:::83:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:8761d0bb:::84:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:9ef0186f:::85:head 2024-09-16T09:04:37.613 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:e0d41294:::86:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:961de695:::87:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:1423148f:::88:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:633a8fa2:::89:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:a8653809:::90:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:3dac8b33:::91:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:35aad435:::92:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:f6dcc343:::93:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:dbbdad87:::94:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:1cb48ce0:::95:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:03cd461c:::96:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:17a4ea99:::97:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:9993c9a7:::98:head 2024-09-16T09:04:37.614 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:6394211c:::99:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:94c7ae57:::100:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:6fdee5bb:::101:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:9a477fd1:::102:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:eb850916:::103:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:affc56b9:::104:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:b42dc814:::105:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:f319f8f0:::106:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:9a40b9de:::107:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:8b524f28:::108:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:e3de589f:::109:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:90f90a5b:::110:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:a7b4f1d7:::111:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:af51766e:::112:head 2024-09-16T09:04:37.615 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:b6f90bd1:::113:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:e0261208:::114:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c9569ef7:::115:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:61bebe50:::116:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:fe93412b:::117:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d3d38bee:::118:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:3100ba0c:::119:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d0560ada:::120:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:f0ea8b35:::121:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:766f231a:::122:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:a07a2582:::123:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:bd7c6b3a:::124:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:fb2ddaff:::125:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:4408e1fe:::126:head 2024-09-16T09:04:37.616 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:ee1df7a7:::127:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c3002909:::128:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:4f48ffa9:::129:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:edf38733:::130:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c08425c0:::131:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:5f902d98:::132:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:41ea2c93:::133:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:813cee13:::134:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:0131818d:::135:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:26ba5a85:::136:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:381b8a5a:::137:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:28797e47:::138:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:bfca7f22:::139:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:36807075:::140:head 2024-09-16T09:04:37.617 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:80b03975:::141:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:5c15709b:::142:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:f39ea15e:::143:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:ea992956:::144:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:48887b1c:::145:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:9f24a9dd:::146:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:987f100b:::147:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d2dd3581:::148:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:7fed1808:::149:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c80b70e9:::150:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:85ed90f9:::151:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:36428b24:::152:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d044c34a:::153:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:7c18bf58:::154:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d1c21232:::155:head 2024-09-16T09:04:37.618 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:a7a3c575:::156:head 2024-09-16T09:04:37.619 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:87da0633:::157:head 2024-09-16T09:04:37.619 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d5ac3822:::158:head 2024-09-16T09:04:37.619 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:3f20522d:::159:head 2024-09-16T09:04:37.619 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:6ca26563:::160:head 2024-09-16T09:04:37.619 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:532ce135:::161:head 2024-09-16T09:04:37.619 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:c78863e6:::162:head 2024-09-16T09:04:37.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:04:37.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi111-76450-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:37 smithi114 ceph-mon[33315]: osdmap e287: 8 total, 8 up, 8 in 2024-09-16T09:04:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushPP_smithi111-76450-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_type","val": "explicit_hash"} : dispatch 2024-09-16T09:04:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushPP_smithi111-76450-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_type","val": "explicit_hash"} : dispatch 2024-09-16T09:04:37.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:37 smithi114 ceph-mon[33315]: pgmap v380: 228 pgs: 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:04:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi111-76450-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[40056]: osdmap e287: 8 total, 8 up, 8 in 2024-09-16T09:04:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushPP_smithi111-76450-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_type","val": "explicit_hash"} : dispatch 2024-09-16T09:04:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushPP_smithi111-76450-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_type","val": "explicit_hash"} : dispatch 2024-09-16T09:04:37.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[40056]: pgmap v380: 228 pgs: 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:04:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi111-76450-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[29803]: osdmap e287: 8 total, 8 up, 8 in 2024-09-16T09:04:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushPP_smithi111-76450-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_type","val": "explicit_hash"} : dispatch 2024-09-16T09:04:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushPP_smithi111-76450-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_type","val": "explicit_hash"} : dispatch 2024-09-16T09:04:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:37 smithi111 ceph-mon[29803]: pgmap v380: 228 pgs: 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:38.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-09-16T09:04:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:38 smithi114 ceph-mon[33315]: osdmap e288: 8 total, 8 up, 8 in 2024-09-16T09:04:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool get","pool":"test-rados-api-smithi111-76716-43","var": "pg_num","format": "json"} : dispatch 2024-09-16T09:04:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43"} : dispatch 2024-09-16T09:04:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43"} : dispatch 2024-09-16T09:04:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-09-16T09:04:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[29803]: osdmap e288: 8 total, 8 up, 8 in 2024-09-16T09:04:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool get","pool":"test-rados-api-smithi111-76716-43","var": "pg_num","format": "json"} : dispatch 2024-09-16T09:04:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43"} : dispatch 2024-09-16T09:04:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43"} : dispatch 2024-09-16T09:04:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-09-16T09:04:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[40056]: osdmap e288: 8 total, 8 up, 8 in 2024-09-16T09:04:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool get","pool":"test-rados-api-smithi111-76716-43","var": "pg_num","format": "json"} : dispatch 2024-09-16T09:04:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43"} : dispatch 2024-09-16T09:04:38.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43"} : dispatch 2024-09-16T09:04:39.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:39 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi111-76450-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi111-76450-57"}]': finished 2024-09-16T09:04:39.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:39 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43"}]': finished 2024-09-16T09:04:39.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:39 smithi114 ceph-mon[33315]: osdmap e289: 8 total, 8 up, 8 in 2024-09-16T09:04:39.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:39 smithi114 ceph-mon[33315]: pgmap v383: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:39.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:39 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi111-76450-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi111-76450-57"}]': finished 2024-09-16T09:04:39.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:39 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43"}]': finished 2024-09-16T09:04:39.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:39 smithi111 ceph-mon[29803]: osdmap e289: 8 total, 8 up, 8 in 2024-09-16T09:04:39.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:39 smithi111 ceph-mon[29803]: pgmap v383: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:39.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:39 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi111-76450-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi111-76450-57"}]': finished 2024-09-16T09:04:39.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:39 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-43"}]': finished 2024-09-16T09:04:39.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:39 smithi111 ceph-mon[40056]: osdmap e289: 8 total, 8 up, 8 in 2024-09-16T09:04:39.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:39 smithi111 ceph-mon[40056]: pgmap v383: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:40.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:40 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:40 smithi114 ceph-mon[33315]: osdmap e290: 8 total, 8 up, 8 in 2024-09-16T09:04:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:40 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:40 smithi111 ceph-mon[29803]: osdmap e290: 8 total, 8 up, 8 in 2024-09-16T09:04:40.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:40 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:40.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:40 smithi111 ceph-mon[40056]: osdmap e290: 8 total, 8 up, 8 in 2024-09-16T09:04:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:41 smithi114 ceph-mon[33315]: osdmap e291: 8 total, 8 up, 8 in 2024-09-16T09:04:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:41 smithi114 ceph-mon[33315]: pgmap v386: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[29803]: osdmap e291: 8 total, 8 up, 8 in 2024-09-16T09:04:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[29803]: pgmap v386: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[40056]: osdmap e291: 8 total, 8 up, 8 in 2024-09-16T09:04:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:41 smithi111 ceph-mon[40056]: pgmap v386: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:42.163 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:04:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:04:42.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"}]': finished 2024-09-16T09:04:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:42 smithi114 ceph-mon[33315]: osdmap e292: 8 total, 8 up, 8 in 2024-09-16T09:04:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"}]': finished 2024-09-16T09:04:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[29803]: osdmap e292: 8 total, 8 up, 8 in 2024-09-16T09:04:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi111-76450-57"}]': finished 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[40056]: osdmap e292: 8 total, 8 up, 8 in 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1461391726' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"} : dispatch 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:04:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:04:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:04:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"}]': finished 2024-09-16T09:04:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[29803]: osdmap e293: 8 total, 8 up, 8 in 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_count","val": "3"} : dispatch 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_count","val": "3"} : dispatch 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi111-76450-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[29803]: pgmap v389: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"}]': finished 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[40056]: osdmap e293: 8 total, 8 up, 8 in 2024-09-16T09:04:43.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_count","val": "3"} : dispatch 2024-09-16T09:04:43.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_count","val": "3"} : dispatch 2024-09-16T09:04:43.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:43.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:43.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi111-76450-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:43.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:43 smithi111 ceph-mon[40056]: pgmap v389: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:43.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi111-76450-57"}]': finished 2024-09-16T09:04:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:04:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:43 smithi114 ceph-mon[33315]: osdmap e293: 8 total, 8 up, 8 in 2024-09-16T09:04:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_count","val": "3"} : dispatch 2024-09-16T09:04:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_count","val": "3"} : dispatch 2024-09-16T09:04:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi111-76450-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:43 smithi114 ceph-mon[33315]: pgmap v389: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:44.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_count","val": "3"}]': finished 2024-09-16T09:04:44.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi111-76450-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:44.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_period","val": "3"} : dispatch 2024-09-16T09:04:44.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:44 smithi114 ceph-mon[33315]: osdmap e294: 8 total, 8 up, 8 in 2024-09-16T09:04:44.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi111-76450-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:44.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_period","val": "3"} : dispatch 2024-09-16T09:04:44.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_count","val": "3"}]': finished 2024-09-16T09:04:44.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi111-76450-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:44.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_period","val": "3"} : dispatch 2024-09-16T09:04:44.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[29803]: osdmap e294: 8 total, 8 up, 8 in 2024-09-16T09:04:44.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi111-76450-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:44.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_period","val": "3"} : dispatch 2024-09-16T09:04:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_count","val": "3"}]': finished 2024-09-16T09:04:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi111-76450-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_period","val": "3"} : dispatch 2024-09-16T09:04:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[40056]: osdmap e294: 8 total, 8 up, 8 in 2024-09-16T09:04:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi111-76450-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_period","val": "3"} : dispatch 2024-09-16T09:04:45.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_period","val": "3"}]': finished 2024-09-16T09:04:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:04:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: osdmap e295: 8 total, 8 up, 8 in 2024-09-16T09:04:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:04:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: pgmap v392: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:04:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi111-76450-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi111-76450-58"}]': finished 2024-09-16T09:04:45.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:04:45.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: osdmap e296: 8 total, 8 up, 8 in 2024-09-16T09:04:45.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_fpp","val": ".01"} : dispatch 2024-09-16T09:04:45.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_fpp","val": ".01"} : dispatch 2024-09-16T09:04:45.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_period","val": "3"}]': finished 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: osdmap e295: 8 total, 8 up, 8 in 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: pgmap v392: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi111-76450-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi111-76450-58"}]': finished 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:04:45.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: osdmap e296: 8 total, 8 up, 8 in 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_fpp","val": ".01"} : dispatch 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_fpp","val": ".01"} : dispatch 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_period","val": "3"}]': finished 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: osdmap e295: 8 total, 8 up, 8 in 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: pgmap v392: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:45.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi111-76450-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi111-76450-58"}]': finished 2024-09-16T09:04:45.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:04:45.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: osdmap e296: 8 total, 8 up, 8 in 2024-09-16T09:04:45.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_fpp","val": ".01"} : dispatch 2024-09-16T09:04:45.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_fpp","val": ".01"} : dispatch 2024-09-16T09:04:46.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:04:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:04:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:46 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-16T09:04:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:46 smithi114 ceph-mon[33315]: osdmap e297: 8 total, 8 up, 8 in 2024-09-16T09:04:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:04:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:04:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-16T09:04:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[29803]: osdmap e297: 8 total, 8 up, 8 in 2024-09-16T09:04:46.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:04:46.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:04:46.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:04:46.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:04:46.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-16T09:04:46.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:46 smithi111 ceph-mon[40056]: osdmap e297: 8 total, 8 up, 8 in 2024-09-16T09:04:47.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:47 smithi114 ceph-mon[33315]: pgmap v395: 236 pgs: 1 creating+activating, 235 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:04:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:47 smithi114 ceph-mon[33315]: osdmap e298: 8 total, 8 up, 8 in 2024-09-16T09:04:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:47.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:47 smithi111 ceph-mon[29803]: pgmap v395: 236 pgs: 1 creating+activating, 235 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:04:47.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:47 smithi111 ceph-mon[29803]: osdmap e298: 8 total, 8 up, 8 in 2024-09-16T09:04:47.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:47.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:47 smithi111 ceph-mon[40056]: pgmap v395: 236 pgs: 1 creating+activating, 235 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:04:47.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:47 smithi111 ceph-mon[40056]: osdmap e298: 8 total, 8 up, 8 in 2024-09-16T09:04:47.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:48.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:48 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:48.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi111-76450-58"}]': finished 2024-09-16T09:04:48.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:48 smithi114 ceph-mon[33315]: osdmap e299: 8 total, 8 up, 8 in 2024-09-16T09:04:48.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:48 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi111-76450-58"}]': finished 2024-09-16T09:04:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:48 smithi111 ceph-mon[29803]: osdmap e299: 8 total, 8 up, 8 in 2024-09-16T09:04:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:48.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:48 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:48.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi111-76450-58"}]': finished 2024-09-16T09:04:48.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:48 smithi111 ceph-mon[40056]: osdmap e299: 8 total, 8 up, 8 in 2024-09-16T09:04:48.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi111-76450-58"} : dispatch 2024-09-16T09:04:49.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:49 smithi114 ceph-mon[33315]: pgmap v398: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:04:49.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi111-76450-58"}]': finished 2024-09-16T09:04:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:49 smithi114 ceph-mon[33315]: osdmap e300: 8 total, 8 up, 8 in 2024-09-16T09:04:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:49 smithi111 ceph-mon[29803]: pgmap v398: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:04:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi111-76450-58"}]': finished 2024-09-16T09:04:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:49 smithi111 ceph-mon[29803]: osdmap e300: 8 total, 8 up, 8 in 2024-09-16T09:04:49.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:49 smithi111 ceph-mon[40056]: pgmap v398: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:04:49.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/826983362' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi111-76450-58"}]': finished 2024-09-16T09:04:49.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:49 smithi111 ceph-mon[40056]: osdmap e300: 8 total, 8 up, 8 in 2024-09-16T09:04:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:50.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:50.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:50.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:04:51.747 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:51.747 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi111-76450-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:51.747 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:51 smithi114 ceph-mon[33315]: osdmap e301: 8 total, 8 up, 8 in 2024-09-16T09:04:51.747 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi111-76450-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:51.747 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:51 smithi114 ceph-mon[33315]: pgmap v401: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi111-76450-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:51.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[40056]: osdmap e301: 8 total, 8 up, 8 in 2024-09-16T09:04:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi111-76450-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[40056]: pgmap v401: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:51.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:04:51.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi111-76450-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:51.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[29803]: osdmap e301: 8 total, 8 up, 8 in 2024-09-16T09:04:51.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi111-76450-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:51.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:51 smithi111 ceph-mon[29803]: pgmap v401: 228 pgs: 228 active+clean; 8.4 MiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:04:52.165 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:04:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:04:52.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:52 smithi114 ceph-mon[33315]: osdmap e302: 8 total, 8 up, 8 in 2024-09-16T09:04:52.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:52 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:52 smithi111 ceph-mon[29803]: osdmap e302: 8 total, 8 up, 8 in 2024-09-16T09:04:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:52 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:52.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:52 smithi111 ceph-mon[40056]: osdmap e302: 8 total, 8 up, 8 in 2024-09-16T09:04:52.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:52 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:04:53.485 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:04:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:04:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:04:53.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi111-76450-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi111-76450-59"}]': finished 2024-09-16T09:04:53.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:53 smithi114 ceph-mon[33315]: osdmap e303: 8 total, 8 up, 8 in 2024-09-16T09:04:53.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:53 smithi114 ceph-mon[33315]: pgmap v404: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-16T09:04:53.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:53 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi111-76450-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi111-76450-59"}]': finished 2024-09-16T09:04:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:53 smithi111 ceph-mon[29803]: osdmap e303: 8 total, 8 up, 8 in 2024-09-16T09:04:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:53 smithi111 ceph-mon[29803]: pgmap v404: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-16T09:04:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:53 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:53.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi111-76450-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi111-76450-59"}]': finished 2024-09-16T09:04:53.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:53 smithi111 ceph-mon[40056]: osdmap e303: 8 total, 8 up, 8 in 2024-09-16T09:04:53.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:53 smithi111 ceph-mon[40056]: pgmap v404: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-16T09:04:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:53 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:04:54.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:54 smithi114 ceph-mon[33315]: osdmap e304: 8 total, 8 up, 8 in 2024-09-16T09:04:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:54 smithi111 ceph-mon[29803]: osdmap e304: 8 total, 8 up, 8 in 2024-09-16T09:04:54.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:54 smithi111 ceph-mon[40056]: osdmap e304: 8 total, 8 up, 8 in 2024-09-16T09:04:55.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:55 smithi114 ceph-mon[33315]: osdmap e305: 8 total, 8 up, 8 in 2024-09-16T09:04:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:55 smithi114 ceph-mon[33315]: pgmap v407: 228 pgs: 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-16T09:04:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"}]': finished 2024-09-16T09:04:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:55 smithi114 ceph-mon[33315]: osdmap e306: 8 total, 8 up, 8 in 2024-09-16T09:04:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[29803]: osdmap e305: 8 total, 8 up, 8 in 2024-09-16T09:04:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[29803]: pgmap v407: 228 pgs: 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-16T09:04:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"}]': finished 2024-09-16T09:04:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[29803]: osdmap e306: 8 total, 8 up, 8 in 2024-09-16T09:04:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[40056]: osdmap e305: 8 total, 8 up, 8 in 2024-09-16T09:04:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[40056]: pgmap v407: 228 pgs: 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-16T09:04:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi111-76450-59"}]': finished 2024-09-16T09:04:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:55 smithi111 ceph-mon[40056]: osdmap e306: 8 total, 8 up, 8 in 2024-09-16T09:04:56.495 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:7462ddf6:::.RoundTripAppendPP (3015 ms) 2024-09-16T09:04:56.495 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3019 ms) 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2958 ms) 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3030 ms) 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: setting pool EIO 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: max_success 108, min_failed 109 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4024 ms) 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3024 ms) 2024-09-16T09:04:56.496 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (114466 ms total) 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2999 ms) 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9031 ms) 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14918 ms) 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3035 ms) 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29983 ms total) 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: 2024-09-16T09:04:56.497 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3002 ms) 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3002 ms total) 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13964 ms) 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7006 ms) 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7027 ms) 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7031 ms) 2024-09-16T09:04:56.498 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3011 ms) 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7063 ms) 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6877 ms) 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7043 ms) 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7023 ms) 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7024 ms) 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7056 ms) 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-16T09:04:56.499 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7009 ms) 2024-09-16T09:04:56.500 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-16T09:04:56.500 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7065 ms) 2024-09-16T09:04:56.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:56.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:56.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"}]': finished 2024-09-16T09:04:56.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:56 smithi114 ceph-mon[33315]: osdmap e307: 8 total, 8 up, 8 in 2024-09-16T09:04:56.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:56.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:56.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"}]': finished 2024-09-16T09:04:56.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:56 smithi111 ceph-mon[29803]: osdmap e307: 8 total, 8 up, 8 in 2024-09-16T09:04:56.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1581908347' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:56.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"} : dispatch 2024-09-16T09:04:56.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi111-76450-59"}]': finished 2024-09-16T09:04:56.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:56 smithi111 ceph-mon[40056]: osdmap e307: 8 total, 8 up, 8 in 2024-09-16T09:04:57.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:57 smithi114 ceph-mon[33315]: pgmap v410: 228 pgs: 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-16T09:04:57.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:57 smithi111 ceph-mon[29803]: pgmap v410: 228 pgs: 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-16T09:04:57.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:57 smithi111 ceph-mon[40056]: pgmap v410: 228 pgs: 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-16T09:04:58.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:58 smithi114 ceph-mon[33315]: osdmap e308: 8 total, 8 up, 8 in 2024-09-16T09:04:58.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1146974736' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-60","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:58.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-60","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:58.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:58.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:58.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45"} : dispatch 2024-09-16T09:04:58.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45"} : dispatch 2024-09-16T09:04:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[29803]: osdmap e308: 8 total, 8 up, 8 in 2024-09-16T09:04:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1146974736' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-60","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-60","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45"} : dispatch 2024-09-16T09:04:58.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45"} : dispatch 2024-09-16T09:04:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[40056]: osdmap e308: 8 total, 8 up, 8 in 2024-09-16T09:04:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1146974736' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-60","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-60","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:04:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:04:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45"} : dispatch 2024-09-16T09:04:58.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45"} : dispatch 2024-09-16T09:04:59.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:59.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45"}]': finished 2024-09-16T09:04:59.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:59 smithi114 ceph-mon[33315]: osdmap e309: 8 total, 8 up, 8 in 2024-09-16T09:04:59.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:04:59 smithi114 ceph-mon[33315]: pgmap v413: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-16T09:04:59.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:59.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45"}]': finished 2024-09-16T09:04:59.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:59 smithi111 ceph-mon[29803]: osdmap e309: 8 total, 8 up, 8 in 2024-09-16T09:04:59.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:04:59 smithi111 ceph-mon[29803]: pgmap v413: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-16T09:04:59.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi111-76450-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:04:59.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-45"}]': finished 2024-09-16T09:04:59.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:59 smithi111 ceph-mon[40056]: osdmap e309: 8 total, 8 up, 8 in 2024-09-16T09:04:59.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:04:59 smithi111 ceph-mon[40056]: pgmap v413: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-16T09:05:00.827 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[29803]: osdmap e310: 8 total, 8 up, 8 in 2024-09-16T09:05:00.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi111-76450-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:00.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi111-76450-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:00.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[40056]: osdmap e310: 8 total, 8 up, 8 in 2024-09-16T09:05:00.829 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.829 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.829 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.829 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.829 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi111-76450-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:00.829 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi111-76450-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:00.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:00 smithi114 ceph-mon[33315]: osdmap e310: 8 total, 8 up, 8 in 2024-09-16T09:05:00.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:00.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi111-76450-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:00.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi111-76450-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:01.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:05:01 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:05:01.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi111-76450-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:01.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:01 smithi114 ceph-mon[33315]: osdmap e311: 8 total, 8 up, 8 in 2024-09-16T09:05:01.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPP_smithi111-76450-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:01.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPP_smithi111-76450-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:01.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:01.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:01.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:01 smithi114 ceph-mon[33315]: pgmap v416: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:01.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi111-76450-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[29803]: osdmap e311: 8 total, 8 up, 8 in 2024-09-16T09:05:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPP_smithi111-76450-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPP_smithi111-76450-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:01.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:01.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[29803]: pgmap v416: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:01.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:01.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi111-76450-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:01.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[40056]: osdmap e311: 8 total, 8 up, 8 in 2024-09-16T09:05:01.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPP_smithi111-76450-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:01.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPP_smithi111-76450-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:01.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:01.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:01.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[40056]: pgmap v416: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 589 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:01.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:02 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:02 smithi114 ceph-mon[33315]: osdmap e312: 8 total, 8 up, 8 in 2024-09-16T09:05:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[29803]: osdmap e312: 8 total, 8 up, 8 in 2024-09-16T09:05:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:02.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:02.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:02.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:02.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[40056]: osdmap e312: 8 total, 8 up, 8 in 2024-09-16T09:05:02.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:02.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:02.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:03.559 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:05:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:05:03.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi111-76450-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi111-76450-61"}]': finished 2024-09-16T09:05:03.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:05:03.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:03 smithi114 ceph-mon[33315]: pgmap v419: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:03.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:03.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:03 smithi114 ceph-mon[33315]: osdmap e313: 8 total, 8 up, 8 in 2024-09-16T09:05:03.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:03.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi111-76450-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi111-76450-61"}]': finished 2024-09-16T09:05:03.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:05:03.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[40056]: pgmap v419: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:03.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:03.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[40056]: osdmap e313: 8 total, 8 up, 8 in 2024-09-16T09:05:03.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:03.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi111-76450-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi111-76450-61"}]': finished 2024-09-16T09:05:03.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:05:03.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[29803]: pgmap v419: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:03.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:03.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[29803]: osdmap e313: 8 total, 8 up, 8 in 2024-09-16T09:05:03.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:04.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-47"}]': finished 2024-09-16T09:05:04.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-47", "mode": "writeback"} : dispatch 2024-09-16T09:05:04.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:04 smithi114 ceph-mon[33315]: osdmap e314: 8 total, 8 up, 8 in 2024-09-16T09:05:04.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-47", "mode": "writeback"} : dispatch 2024-09-16T09:05:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-47"}]': finished 2024-09-16T09:05:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-47", "mode": "writeback"} : dispatch 2024-09-16T09:05:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:04 smithi111 ceph-mon[29803]: osdmap e314: 8 total, 8 up, 8 in 2024-09-16T09:05:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-47", "mode": "writeback"} : dispatch 2024-09-16T09:05:04.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-47"}]': finished 2024-09-16T09:05:04.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-47", "mode": "writeback"} : dispatch 2024-09-16T09:05:04.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:04 smithi111 ceph-mon[40056]: osdmap e314: 8 total, 8 up, 8 in 2024-09-16T09:05:04.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-47", "mode": "writeback"} : dispatch 2024-09-16T09:05:05.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: pgmap v421: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-47", "mode": "writeback"}]': finished 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: osdmap e315: 8 total, 8 up, 8 in 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"}]': finished 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: osdmap e316: 8 total, 8 up, 8 in 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:05.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: pgmap v421: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:05.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:05:05.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-47", "mode": "writeback"}]': finished 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: osdmap e315: 8 total, 8 up, 8 in 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"}]': finished 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: osdmap e316: 8 total, 8 up, 8 in 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: pgmap v421: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-47", "mode": "writeback"}]': finished 2024-09-16T09:05:05.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: osdmap e315: 8 total, 8 up, 8 in 2024-09-16T09:05:05.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:05.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:05.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:05.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:05.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:05:05.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi111-76450-61"}]': finished 2024-09-16T09:05:05.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: osdmap e316: 8 total, 8 up, 8 in 2024-09-16T09:05:05.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:05.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:06.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:05:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"}]': finished 2024-09-16T09:05:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:06 smithi114 ceph-mon[33315]: osdmap e317: 8 total, 8 up, 8 in 2024-09-16T09:05:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:05:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"}]': finished 2024-09-16T09:05:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[29803]: osdmap e317: 8 total, 8 up, 8 in 2024-09-16T09:05:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:06.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4011515849' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:06.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"} : dispatch 2024-09-16T09:05:06.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:05:06.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi111-76450-61"}]': finished 2024-09-16T09:05:06.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[40056]: osdmap e317: 8 total, 8 up, 8 in 2024-09-16T09:05:06.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:06.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: pgmap v424: 228 pgs: 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:05:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:07.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: osdmap e318: 8 total, 8 up, 8 in 2024-09-16T09:05:07.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi111-76450-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi111-76450-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: pgmap v424: 228 pgs: 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:05:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: osdmap e318: 8 total, 8 up, 8 in 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi111-76450-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi111-76450-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: pgmap v424: 228 pgs: 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:05:07.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: osdmap e318: 8 total, 8 up, 8 in 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi111-76450-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:07.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi111-76450-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:08.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:05:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_grade_decay_rate","val": "20"} : dispatch 2024-09-16T09:05:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:08 smithi114 ceph-mon[33315]: osdmap e319: 8 total, 8 up, 8 in 2024-09-16T09:05:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_grade_decay_rate","val": "20"} : dispatch 2024-09-16T09:05:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:05:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_grade_decay_rate","val": "20"} : dispatch 2024-09-16T09:05:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[29803]: osdmap e319: 8 total, 8 up, 8 in 2024-09-16T09:05:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_grade_decay_rate","val": "20"} : dispatch 2024-09-16T09:05:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:05:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_grade_decay_rate","val": "20"} : dispatch 2024-09-16T09:05:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[40056]: osdmap e319: 8 total, 8 up, 8 in 2024-09-16T09:05:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_grade_decay_rate","val": "20"} : dispatch 2024-09-16T09:05:09.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:09 smithi114 ceph-mon[33315]: pgmap v427: 228 pgs: 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:05:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi111-76450-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi111-76450-62"}]': finished 2024-09-16T09:05:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-16T09:05:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:09 smithi114 ceph-mon[33315]: osdmap e320: 8 total, 8 up, 8 in 2024-09-16T09:05:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_search_last_n","val": "1"} : dispatch 2024-09-16T09:05:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_search_last_n","val": "1"} : dispatch 2024-09-16T09:05:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[29803]: pgmap v427: 228 pgs: 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:05:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi111-76450-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi111-76450-62"}]': finished 2024-09-16T09:05:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-16T09:05:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[29803]: osdmap e320: 8 total, 8 up, 8 in 2024-09-16T09:05:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_search_last_n","val": "1"} : dispatch 2024-09-16T09:05:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_search_last_n","val": "1"} : dispatch 2024-09-16T09:05:09.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[40056]: pgmap v427: 228 pgs: 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:05:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi111-76450-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi111-76450-62"}]': finished 2024-09-16T09:05:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-16T09:05:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[40056]: osdmap e320: 8 total, 8 up, 8 in 2024-09-16T09:05:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_search_last_n","val": "1"} : dispatch 2024-09-16T09:05:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_search_last_n","val": "1"} : dispatch 2024-09-16T09:05:11.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:11 smithi114 ceph-mon[33315]: pgmap v430: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:11.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-16T09:05:11.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:11 smithi114 ceph-mon[33315]: osdmap e321: 8 total, 8 up, 8 in 2024-09-16T09:05:11.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:11.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:11.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:05:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:05:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[29803]: pgmap v430: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-16T09:05:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[29803]: osdmap e321: 8 total, 8 up, 8 in 2024-09-16T09:05:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:11.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[40056]: pgmap v430: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-16T09:05:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[40056]: osdmap e321: 8 total, 8 up, 8 in 2024-09-16T09:05:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:12.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:05:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:12 smithi114 ceph-mon[33315]: osdmap e322: 8 total, 8 up, 8 in 2024-09-16T09:05:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:12 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:05:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[29803]: osdmap e322: 8 total, 8 up, 8 in 2024-09-16T09:05:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[40056]: osdmap e322: 8 total, 8 up, 8 in 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:12 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:13.604 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:05:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:05:13.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: pgmap v433: 228 pgs: 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"}]': finished 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"}]': finished 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: osdmap e323: 8 total, 8 up, 8 in 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: pgmap v433: 228 pgs: 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:05:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"}]': finished 2024-09-16T09:05:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"}]': finished 2024-09-16T09:05:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: osdmap e323: 8 total, 8 up, 8 in 2024-09-16T09:05:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: pgmap v433: 228 pgs: 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi111-76450-62"}]': finished 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"}]': finished 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: osdmap e323: 8 total, 8 up, 8 in 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4198095928' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"} : dispatch 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:13.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:13.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-47"} : dispatch 2024-09-16T09:05:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"}]': finished 2024-09-16T09:05:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:14 smithi114 ceph-mon[33315]: osdmap e324: 8 total, 8 up, 8 in 2024-09-16T09:05:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi111-76450-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi111-76450-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:14 smithi114 ceph-mon[33315]: osdmap e325: 8 total, 8 up, 8 in 2024-09-16T09:05:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "StatRemovePP_smithi111-76450-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-49","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"}]': finished 2024-09-16T09:05:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[29803]: osdmap e324: 8 total, 8 up, 8 in 2024-09-16T09:05:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi111-76450-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi111-76450-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[29803]: osdmap e325: 8 total, 8 up, 8 in 2024-09-16T09:05:14.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "StatRemovePP_smithi111-76450-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-49","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi111-76450-62"}]': finished 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[40056]: osdmap e324: 8 total, 8 up, 8 in 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi111-76450-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi111-76450-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[40056]: osdmap e325: 8 total, 8 up, 8 in 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "StatRemovePP_smithi111-76450-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:14.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-49","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:15 smithi114 ceph-mon[33315]: pgmap v436: 196 pgs: 196 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:05:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-49","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:15.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:15 smithi111 ceph-mon[29803]: pgmap v436: 196 pgs: 196 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:05:15.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-49","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:15.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:15 smithi111 ceph-mon[40056]: pgmap v436: 196 pgs: 196 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:05:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-49","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:16.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:16 smithi114 ceph-mon[33315]: osdmap e326: 8 total, 8 up, 8 in 2024-09-16T09:05:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi111-76450-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi111-76450-63"}]': finished 2024-09-16T09:05:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:05:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:16 smithi114 ceph-mon[33315]: osdmap e327: 8 total, 8 up, 8 in 2024-09-16T09:05:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:16.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[29803]: osdmap e326: 8 total, 8 up, 8 in 2024-09-16T09:05:16.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:16.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:16.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:16.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi111-76450-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi111-76450-63"}]': finished 2024-09-16T09:05:16.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[29803]: osdmap e327: 8 total, 8 up, 8 in 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[40056]: osdmap e326: 8 total, 8 up, 8 in 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi111-76450-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi111-76450-63"}]': finished 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[40056]: osdmap e327: 8 total, 8 up, 8 in 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:17.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:17 smithi114 ceph-mon[33315]: pgmap v439: 228 pgs: 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:05:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-49"}]': finished 2024-09-16T09:05:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:17 smithi114 ceph-mon[33315]: osdmap e328: 8 total, 8 up, 8 in 2024-09-16T09:05:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-49", "mode": "readproxy"} : dispatch 2024-09-16T09:05:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-49", "mode": "readproxy"} : dispatch 2024-09-16T09:05:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[29803]: pgmap v439: 228 pgs: 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:05:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-49"}]': finished 2024-09-16T09:05:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[29803]: osdmap e328: 8 total, 8 up, 8 in 2024-09-16T09:05:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-49", "mode": "readproxy"} : dispatch 2024-09-16T09:05:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-49", "mode": "readproxy"} : dispatch 2024-09-16T09:05:17.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[40056]: pgmap v439: 228 pgs: 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:05:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-49"}]': finished 2024-09-16T09:05:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[40056]: osdmap e328: 8 total, 8 up, 8 in 2024-09-16T09:05:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-49", "mode": "readproxy"} : dispatch 2024-09-16T09:05:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-49", "mode": "readproxy"} : dispatch 2024-09-16T09:05:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:18 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:05:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-49", "mode": "readproxy"}]': finished 2024-09-16T09:05:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:18 smithi114 ceph-mon[33315]: osdmap e329: 8 total, 8 up, 8 in 2024-09-16T09:05:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:18 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:05:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-49", "mode": "readproxy"}]': finished 2024-09-16T09:05:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:18 smithi111 ceph-mon[29803]: osdmap e329: 8 total, 8 up, 8 in 2024-09-16T09:05:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:18.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:18 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:05:18.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-49", "mode": "readproxy"}]': finished 2024-09-16T09:05:18.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:18 smithi111 ceph-mon[40056]: osdmap e329: 8 total, 8 up, 8 in 2024-09-16T09:05:18.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:19.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:19 smithi114 ceph-mon[33315]: pgmap v442: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:05:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:19 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi111-76450-63"}]': finished 2024-09-16T09:05:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:19 smithi114 ceph-mon[33315]: osdmap e330: 8 total, 8 up, 8 in 2024-09-16T09:05:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[29803]: pgmap v442: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:05:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi111-76450-63"}]': finished 2024-09-16T09:05:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[29803]: osdmap e330: 8 total, 8 up, 8 in 2024-09-16T09:05:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[40056]: pgmap v442: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:05:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi111-76450-63"}]': finished 2024-09-16T09:05:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[40056]: osdmap e330: 8 total, 8 up, 8 in 2024-09-16T09:05:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi111-76450-63"} : dispatch 2024-09-16T09:05:21.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:05:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-mon[33315]: pgmap v445: 228 pgs: 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi111-76450-63"}]': finished 2024-09-16T09:05:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-mon[33315]: osdmap e331: 8 total, 8 up, 8 in 2024-09-16T09:05:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi111-76450-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi111-76450-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[29803]: pgmap v445: 228 pgs: 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi111-76450-63"}]': finished 2024-09-16T09:05:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[29803]: osdmap e331: 8 total, 8 up, 8 in 2024-09-16T09:05:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi111-76450-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi111-76450-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[40056]: pgmap v445: 228 pgs: 228 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3856257182' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi111-76450-63"}]': finished 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[40056]: osdmap e331: 8 total, 8 up, 8 in 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi111-76450-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi111-76450-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi111-76450-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:22 smithi114 ceph-mon[33315]: osdmap e332: 8 total, 8 up, 8 in 2024-09-16T09:05:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi111-76450-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi111-76450-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:22.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi111-76450-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[29803]: osdmap e332: 8 total, 8 up, 8 in 2024-09-16T09:05:22.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi111-76450-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:22.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi111-76450-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:22.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi111-76450-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[40056]: osdmap e332: 8 total, 8 up, 8 in 2024-09-16T09:05:22.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi111-76450-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:22.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi111-76450-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:22.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:23.634 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:05:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:05:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:05:23.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:23 smithi114 ceph-mon[33315]: pgmap v448: 228 pgs: 228 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-16T09:05:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:23 smithi114 ceph-mon[33315]: osdmap e333: 8 total, 8 up, 8 in 2024-09-16T09:05:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:23 smithi111 ceph-mon[29803]: pgmap v448: 228 pgs: 228 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-16T09:05:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:23 smithi111 ceph-mon[29803]: osdmap e333: 8 total, 8 up, 8 in 2024-09-16T09:05:23.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:23 smithi111 ceph-mon[40056]: pgmap v448: 228 pgs: 228 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-16T09:05:23.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:23 smithi111 ceph-mon[40056]: osdmap e333: 8 total, 8 up, 8 in 2024-09-16T09:05:24.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:24 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi111-76450-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi111-76450-64"}]': finished 2024-09-16T09:05:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:24 smithi114 ceph-mon[33315]: osdmap e334: 8 total, 8 up, 8 in 2024-09-16T09:05:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:24 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi111-76450-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi111-76450-64"}]': finished 2024-09-16T09:05:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:24 smithi111 ceph-mon[29803]: osdmap e334: 8 total, 8 up, 8 in 2024-09-16T09:05:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:24 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi111-76450-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi111-76450-64"}]': finished 2024-09-16T09:05:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:24 smithi111 ceph-mon[40056]: osdmap e334: 8 total, 8 up, 8 in 2024-09-16T09:05:25.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:25 smithi114 ceph-mon[33315]: pgmap v451: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-16T09:05:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:25 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:25 smithi114 ceph-mon[33315]: osdmap e335: 8 total, 8 up, 8 in 2024-09-16T09:05:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:25 smithi111 ceph-mon[29803]: pgmap v451: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-16T09:05:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:25 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:25 smithi111 ceph-mon[29803]: osdmap e335: 8 total, 8 up, 8 in 2024-09-16T09:05:25.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:25 smithi111 ceph-mon[40056]: pgmap v451: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-16T09:05:25.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:25 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:25.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:25 smithi111 ceph-mon[40056]: osdmap e335: 8 total, 8 up, 8 in 2024-09-16T09:05:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:26 smithi114 ceph-mon[33315]: osdmap e336: 8 total, 8 up, 8 in 2024-09-16T09:05:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:26 smithi111 ceph-mon[29803]: osdmap e336: 8 total, 8 up, 8 in 2024-09-16T09:05:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:26 smithi111 ceph-mon[40056]: osdmap e336: 8 total, 8 up, 8 in 2024-09-16T09:05:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[29803]: pgmap v454: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"}]': finished 2024-09-16T09:05:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[29803]: osdmap e337: 8 total, 8 up, 8 in 2024-09-16T09:05:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[40056]: pgmap v454: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"}]': finished 2024-09-16T09:05:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[40056]: osdmap e337: 8 total, 8 up, 8 in 2024-09-16T09:05:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:28.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:27 smithi114 ceph-mon[33315]: pgmap v454: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi111-76450-64"}]': finished 2024-09-16T09:05:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:27 smithi114 ceph-mon[33315]: osdmap e337: 8 total, 8 up, 8 in 2024-09-16T09:05:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/398781399' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"} : dispatch 2024-09-16T09:05:29.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"}]': finished 2024-09-16T09:05:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:28 smithi114 ceph-mon[33315]: osdmap e338: 8 total, 8 up, 8 in 2024-09-16T09:05:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi111-76450-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi111-76450-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"}]': finished 2024-09-16T09:05:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[29803]: osdmap e338: 8 total, 8 up, 8 in 2024-09-16T09:05:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi111-76450-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:29.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi111-76450-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi111-76450-64"}]': finished 2024-09-16T09:05:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[40056]: osdmap e338: 8 total, 8 up, 8 in 2024-09-16T09:05:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi111-76450-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:29.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi111-76450-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:30.077 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[29803]: pgmap v457: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:30.078 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi111-76450-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:30.078 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[29803]: osdmap e339: 8 total, 8 up, 8 in 2024-09-16T09:05:30.078 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "OmapPP_smithi111-76450-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:30.078 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "OmapPP_smithi111-76450-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:30.078 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:30.078 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:30.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[40056]: pgmap v457: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:30.079 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi111-76450-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:30.079 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[40056]: osdmap e339: 8 total, 8 up, 8 in 2024-09-16T09:05:30.079 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "OmapPP_smithi111-76450-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:30.079 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "OmapPP_smithi111-76450-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:30.079 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:30.079 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:30.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:29 smithi114 ceph-mon[33315]: pgmap v457: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi111-76450-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:29 smithi114 ceph-mon[33315]: osdmap e339: 8 total, 8 up, 8 in 2024-09-16T09:05:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "OmapPP_smithi111-76450-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "OmapPP_smithi111-76450-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:31.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:05:31.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:31.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:30 smithi114 ceph-mon[33315]: osdmap e340: 8 total, 8 up, 8 in 2024-09-16T09:05:31.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:31.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:30 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:05:31.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi111-76450-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi111-76450-65"}]': finished 2024-09-16T09:05:31.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"}]': finished 2024-09-16T09:05:31.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:30 smithi114 ceph-mon[33315]: osdmap e341: 8 total, 8 up, 8 in 2024-09-16T09:05:31.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:05:31.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:31.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[29803]: osdmap e340: 8 total, 8 up, 8 in 2024-09-16T09:05:31.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:31.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:05:31.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi111-76450-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi111-76450-65"}]': finished 2024-09-16T09:05:31.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"}]': finished 2024-09-16T09:05:31.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[29803]: osdmap e341: 8 total, 8 up, 8 in 2024-09-16T09:05:31.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:05:31.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:31.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[40056]: osdmap e340: 8 total, 8 up, 8 in 2024-09-16T09:05:31.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:31.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:05:31.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi111-76450-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi111-76450-65"}]': finished 2024-09-16T09:05:31.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"}]': finished 2024-09-16T09:05:31.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:30 smithi111 ceph-mon[40056]: osdmap e341: 8 total, 8 up, 8 in 2024-09-16T09:05:31.690 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2163:head 2024-09-16T09:05:31.690 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:5d165639:::164:head 2024-09-16T09:05:31.690 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:f43765fc:::165:head 2024-09-16T09:05:31.690 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:b4c720e9:::166:head 2024-09-16T09:05:31.690 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:e694b040:::167:head 2024-09-16T09:05:31.690 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:afa38db2:::168:head 2024-09-16T09:05:31.690 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:77ba9f53:::169:head 2024-09-16T09:05:31.690 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:87495034:::170:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:7c96bf0e:::171:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:dbe346cc:::172:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:e943ec24:::173:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:f97a9c0c:::174:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:6f26e74d:::175:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:4f95e106:::176:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:0e6f2f8f:::177:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:05db05f1:::178:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:38a78d66:::179:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:d095610b:::180:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:a1a9d709:::181:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:1e5d39db:::182:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:f7df4fb9:::183:head 2024-09-16T09:05:31.691 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:03a7f161:::184:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:ba70721e:::185:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:28e5662d:::186:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:973d52de:::187:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:4303eb1c:::188:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:b990b48e:::189:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:29b8165b:::190:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:3547f197:::191:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:7e260936:::192:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:1abec7b1:::193:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:10fdda93:::194:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:15817eea:::195:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:770bab57:::196:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:ed9e13e7:::197:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:71471a8f:::198:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: checking for 276:10fb1d02:::199:head 2024-09-16T09:05:31.692 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8037 ms) 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: first is 1726477487 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,1726477489,1726477490,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,1726477489,1726477490,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,1726477489,1726477490,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,1726477489,1726477490,1726477492,1726477493,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,1726477489,1726477490,1726477492,1726477493,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,1726477489,1726477490,1726477492,1726477493,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,1726477489,1726477490,1726477492,1726477493,1726477495,1726477496,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,1726477489,1726477490,1726477492,1726477493,1726477495,1726477496,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477487,1726477489,1726477490,1726477492,1726477493,1726477495,1726477496,0 2024-09-16T09:05:31.693 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726477490,1726477492,1726477493,1726477495,1726477496,1726477498,1726477499,0 2024-09-16T09:05:31.694 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: first now 1726477490, trimmed 2024-09-16T09:05:31.694 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20143 ms) 2024-09-16T09:05:31.694 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-16T09:05:31.694 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: foo0 2024-09-16T09:05:31.694 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T09:05:31.694 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14060 ms) 2024-09-16T09:05:31.694 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-16T09:05:32.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:05:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:05:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:31 smithi114 ceph-mon[33315]: pgmap v460: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:32.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:32.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:31 smithi114 ceph-mon[33315]: osdmap e342: 8 total, 8 up, 8 in 2024-09-16T09:05:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[29803]: pgmap v460: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:32.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[29803]: osdmap e342: 8 total, 8 up, 8 in 2024-09-16T09:05:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[40056]: pgmap v460: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-49"} : dispatch 2024-09-16T09:05:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:31 smithi111 ceph-mon[40056]: osdmap e342: 8 total, 8 up, 8 in 2024-09-16T09:05:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:32 smithi114 ceph-mon[33315]: osdmap e343: 8 total, 8 up, 8 in 2024-09-16T09:05:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[29803]: osdmap e343: 8 total, 8 up, 8 in 2024-09-16T09:05:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:33.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:33.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:33.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[40056]: osdmap e343: 8 total, 8 up, 8 in 2024-09-16T09:05:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-51","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:05:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:05:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:33 smithi114 ceph-mon[33315]: pgmap v463: 204 pgs: 8 creating+peering, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:33 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"}]': finished 2024-09-16T09:05:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:33 smithi114 ceph-mon[33315]: osdmap e344: 8 total, 8 up, 8 in 2024-09-16T09:05:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[40056]: pgmap v463: 204 pgs: 8 creating+peering, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"}]': finished 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[40056]: osdmap e344: 8 total, 8 up, 8 in 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[29803]: pgmap v463: 204 pgs: 8 creating+peering, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi111-76450-65"}]': finished 2024-09-16T09:05:34.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[29803]: osdmap e344: 8 total, 8 up, 8 in 2024-09-16T09:05:34.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1454243434' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:34.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"} : dispatch 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"}]': finished 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: osdmap e345: 8 total, 8 up, 8 in 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:35.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:35.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"}]': finished 2024-09-16T09:05:35.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:05:35.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:35.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: osdmap e345: 8 total, 8 up, 8 in 2024-09-16T09:05:35.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi111-76450-65"}]': finished 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: osdmap e345: 8 total, 8 up, 8 in 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:35.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: pgmap v466: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi111-76450-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi111-76450-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-51"}]': finished 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi111-76450-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-51", "mode": "writeback"} : dispatch 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "MultiWritePP_smithi111-76450-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: osdmap e346: 8 total, 8 up, 8 in 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-51", "mode": "writeback"} : dispatch 2024-09-16T09:05:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "MultiWritePP_smithi111-76450-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: pgmap v466: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi111-76450-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi111-76450-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-51"}]': finished 2024-09-16T09:05:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi111-76450-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-51", "mode": "writeback"} : dispatch 2024-09-16T09:05:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "MultiWritePP_smithi111-76450-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: osdmap e346: 8 total, 8 up, 8 in 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-51", "mode": "writeback"} : dispatch 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "MultiWritePP_smithi111-76450-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: pgmap v466: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi111-76450-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi111-76450-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-51"}]': finished 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi111-76450-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-51", "mode": "writeback"} : dispatch 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "MultiWritePP_smithi111-76450-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: osdmap e346: 8 total, 8 up, 8 in 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-51", "mode": "writeback"} : dispatch 2024-09-16T09:05:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "MultiWritePP_smithi111-76450-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:37.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:36 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:05:37.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-51", "mode": "writeback"}]': finished 2024-09-16T09:05:37.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:36 smithi114 ceph-mon[33315]: osdmap e347: 8 total, 8 up, 8 in 2024-09-16T09:05:37.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:36 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:05:37.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-51", "mode": "writeback"}]': finished 2024-09-16T09:05:37.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:36 smithi111 ceph-mon[29803]: osdmap e347: 8 total, 8 up, 8 in 2024-09-16T09:05:37.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:36 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:05:37.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-51", "mode": "writeback"}]': finished 2024-09-16T09:05:37.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:36 smithi111 ceph-mon[40056]: osdmap e347: 8 total, 8 up, 8 in 2024-09-16T09:05:38.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:37 smithi114 ceph-mon[33315]: pgmap v469: 228 pgs: 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:05:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi111-76450-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi111-76450-66"}]': finished 2024-09-16T09:05:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:05:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:37 smithi114 ceph-mon[33315]: osdmap e348: 8 total, 8 up, 8 in 2024-09-16T09:05:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[29803]: pgmap v469: 228 pgs: 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:05:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi111-76450-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi111-76450-66"}]': finished 2024-09-16T09:05:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:05:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[29803]: osdmap e348: 8 total, 8 up, 8 in 2024-09-16T09:05:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:38.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[40056]: pgmap v469: 228 pgs: 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:05:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:05:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi111-76450-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi111-76450-66"}]': finished 2024-09-16T09:05:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:05:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[40056]: osdmap e348: 8 total, 8 up, 8 in 2024-09-16T09:05:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:05:39.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:38 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:05:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:38 smithi114 ceph-mon[33315]: osdmap e349: 8 total, 8 up, 8 in 2024-09-16T09:05:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:05:39.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:39.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[29803]: osdmap e349: 8 total, 8 up, 8 in 2024-09-16T09:05:39.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:05:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:05:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[40056]: osdmap e349: 8 total, 8 up, 8 in 2024-09-16T09:05:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:05:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:39 smithi114 ceph-mon[33315]: pgmap v472: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:05:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:39 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:05:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:39 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:05:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:39 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:39 smithi114 ceph-mon[33315]: osdmap e350: 8 total, 8 up, 8 in 2024-09-16T09:05:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:39 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:39 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:39 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[29803]: pgmap v472: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:05:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:05:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:05:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[29803]: osdmap e350: 8 total, 8 up, 8 in 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[40056]: pgmap v472: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[40056]: osdmap e350: 8 total, 8 up, 8 in 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:40.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:39 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:41.722 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ 2024-09-16T09:05:41.722 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3038 ms) 2024-09-16T09:05:41.722 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-16T09:05:41.722 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7037 ms) 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7026 ms) 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7013 ms) 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7072 ms) 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7018 ms) 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7019 ms) 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139422 ms total) 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-16T09:05:41.723 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (286873 ms total) 2024-09-16T09:05:41.724 INFO:tasks.workunit.client.0.smithi111.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-16T09:05:42.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:05:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:05:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:41 smithi114 ceph-mon[33315]: pgmap v475: 228 pgs: 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:05:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"}]': finished 2024-09-16T09:05:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:41 smithi114 ceph-mon[33315]: osdmap e351: 8 total, 8 up, 8 in 2024-09-16T09:05:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "target_max_objects","val": "1"} : dispatch 2024-09-16T09:05:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "target_max_objects","val": "1"} : dispatch 2024-09-16T09:05:42.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:42.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[29803]: pgmap v475: 228 pgs: 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:05:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"}]': finished 2024-09-16T09:05:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[29803]: osdmap e351: 8 total, 8 up, 8 in 2024-09-16T09:05:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "target_max_objects","val": "1"} : dispatch 2024-09-16T09:05:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "target_max_objects","val": "1"} : dispatch 2024-09-16T09:05:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[40056]: pgmap v475: 228 pgs: 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:05:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:05:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi111-76450-66"}]': finished 2024-09-16T09:05:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[40056]: osdmap e351: 8 total, 8 up, 8 in 2024-09-16T09:05:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "target_max_objects","val": "1"} : dispatch 2024-09-16T09:05:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "target_max_objects","val": "1"} : dispatch 2024-09-16T09:05:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1808215799' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"} : dispatch 2024-09-16T09:05:43.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "target_max_objects","val": "1"}]': finished 2024-09-16T09:05:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"}]': finished 2024-09-16T09:05:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:42 smithi114 ceph-mon[33315]: osdmap e352: 8 total, 8 up, 8 in 2024-09-16T09:05:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "target_max_objects","val": "1"}]': finished 2024-09-16T09:05:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"}]': finished 2024-09-16T09:05:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:42 smithi111 ceph-mon[29803]: osdmap e352: 8 total, 8 up, 8 in 2024-09-16T09:05:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-51","var": "target_max_objects","val": "1"}]': finished 2024-09-16T09:05:43.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi111-76450-66"}]': finished 2024-09-16T09:05:43.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:42 smithi111 ceph-mon[40056]: osdmap e352: 8 total, 8 up, 8 in 2024-09-16T09:05:43.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:05:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:05:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:05:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:43 smithi114 ceph-mon[33315]: pgmap v478: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-16T09:05:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:43 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-16T09:05:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:43 smithi111 ceph-mon[29803]: pgmap v478: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-16T09:05:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:43 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-16T09:05:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:43 smithi111 ceph-mon[40056]: pgmap v478: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-16T09:05:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:43 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-16T09:05:46.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:45 smithi114 ceph-mon[33315]: pgmap v479: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:05:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:45 smithi111 ceph-mon[29803]: pgmap v479: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:05:46.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:45 smithi111 ceph-mon[40056]: pgmap v479: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:05:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:05:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:05:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:05:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:05:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:05:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:05:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:05:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:05:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:05:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:05:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:05:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:05:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:05:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:05:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:05:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:05:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:48 smithi114 ceph-mon[33315]: pgmap v480: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-16T09:05:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:48 smithi111 ceph-mon[29803]: pgmap v480: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-16T09:05:48.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:48 smithi111 ceph-mon[40056]: pgmap v480: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-16T09:05:50.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:50 smithi114 ceph-mon[33315]: pgmap v481: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:05:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:50 smithi111 ceph-mon[29803]: pgmap v481: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:05:50.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:50 smithi111 ceph-mon[40056]: pgmap v481: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:05:52.163 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:05:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:05:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:52 smithi114 ceph-mon[33315]: pgmap v482: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:05:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:52 smithi111 ceph-mon[29803]: pgmap v482: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:05:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:52 smithi111 ceph-mon[40056]: pgmap v482: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:05:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:05:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:53 smithi114 ceph-mon[33315]: osdmap e353: 8 total, 8 up, 8 in 2024-09-16T09:05:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"}]': finished 2024-09-16T09:05:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:05:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:05:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:05:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[29803]: osdmap e353: 8 total, 8 up, 8 in 2024-09-16T09:05:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"}]': finished 2024-09-16T09:05:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:05:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:05:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[40056]: osdmap e353: 8 total, 8 up, 8 in 2024-09-16T09:05:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"}]': finished 2024-09-16T09:05:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:54 smithi114 ceph-mon[33315]: pgmap v484: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:05:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:54 smithi114 ceph-mon[33315]: osdmap e354: 8 total, 8 up, 8 in 2024-09-16T09:05:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:54 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:54 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[29803]: pgmap v484: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:05:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[29803]: osdmap e354: 8 total, 8 up, 8 in 2024-09-16T09:05:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[40056]: pgmap v484: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:05:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[40056]: osdmap e354: 8 total, 8 up, 8 in 2024-09-16T09:05:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:54 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-51"} : dispatch 2024-09-16T09:05:55.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:55 smithi114 ceph-mon[33315]: osdmap e355: 8 total, 8 up, 8 in 2024-09-16T09:05:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:55 smithi111 ceph-mon[29803]: osdmap e355: 8 total, 8 up, 8 in 2024-09-16T09:05:55.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:55 smithi111 ceph-mon[40056]: osdmap e355: 8 total, 8 up, 8 in 2024-09-16T09:05:56.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:56 smithi114 ceph-mon[33315]: pgmap v487: 196 pgs: 196 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-16T09:05:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:56 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-16T09:05:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:56 smithi114 ceph-mon[33315]: osdmap e356: 8 total, 8 up, 8 in 2024-09-16T09:05:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-53","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-53","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[29803]: pgmap v487: 196 pgs: 196 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-16T09:05:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-16T09:05:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[29803]: osdmap e356: 8 total, 8 up, 8 in 2024-09-16T09:05:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-53","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-53","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[40056]: pgmap v487: 196 pgs: 196 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-16T09:05:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-16T09:05:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[40056]: osdmap e356: 8 total, 8 up, 8 in 2024-09-16T09:05:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-53","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-53","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:57.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:57 smithi114 ceph-mon[33315]: osdmap e357: 8 total, 8 up, 8 in 2024-09-16T09:05:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-53"} : dispatch 2024-09-16T09:05:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-53"} : dispatch 2024-09-16T09:05:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[29803]: osdmap e357: 8 total, 8 up, 8 in 2024-09-16T09:05:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-53"} : dispatch 2024-09-16T09:05:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-53"} : dispatch 2024-09-16T09:05:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:05:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[40056]: osdmap e357: 8 total, 8 up, 8 in 2024-09-16T09:05:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:05:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-53"} : dispatch 2024-09-16T09:05:57.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-53"} : dispatch 2024-09-16T09:05:58.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:58 smithi114 ceph-mon[33315]: pgmap v490: 228 pgs: 228 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:05:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:58 smithi114 ceph-mon[33315]: osdmap e358: 8 total, 8 up, 8 in 2024-09-16T09:05:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:58 smithi111 ceph-mon[29803]: pgmap v490: 228 pgs: 228 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:05:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:58 smithi111 ceph-mon[29803]: osdmap e358: 8 total, 8 up, 8 in 2024-09-16T09:05:58.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:58 smithi111 ceph-mon[40056]: pgmap v490: 228 pgs: 228 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:05:58.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:58 smithi111 ceph-mon[40056]: osdmap e358: 8 total, 8 up, 8 in 2024-09-16T09:05:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:59 smithi114 ceph-mon[33315]: osdmap e359: 8 total, 8 up, 8 in 2024-09-16T09:05:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-55","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:59.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:05:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-55","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:59 smithi111 ceph-mon[29803]: osdmap e359: 8 total, 8 up, 8 in 2024-09-16T09:05:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-55","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:59.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:05:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-55","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:59 smithi111 ceph-mon[40056]: osdmap e359: 8 total, 8 up, 8 in 2024-09-16T09:05:59.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-55","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:05:59.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:05:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-55","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:00 smithi114 ceph-mon[33315]: pgmap v493: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:06:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:00 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:06:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:00 smithi114 ceph-mon[33315]: osdmap e360: 8 total, 8 up, 8 in 2024-09-16T09:06:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-55"} : dispatch 2024-09-16T09:06:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-55"} : dispatch 2024-09-16T09:06:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[29803]: pgmap v493: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:06:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:06:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[29803]: osdmap e360: 8 total, 8 up, 8 in 2024-09-16T09:06:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-55"} : dispatch 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-55"} : dispatch 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[40056]: pgmap v493: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[40056]: osdmap e360: 8 total, 8 up, 8 in 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-55"} : dispatch 2024-09-16T09:06:00.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-55"} : dispatch 2024-09-16T09:06:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:01 smithi114 ceph-mon[33315]: osdmap e361: 8 total, 8 up, 8 in 2024-09-16T09:06:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:01 smithi114 ceph-mon[33315]: pgmap v496: 196 pgs: 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:06:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:01 smithi111 ceph-mon[29803]: osdmap e361: 8 total, 8 up, 8 in 2024-09-16T09:06:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:01 smithi111 ceph-mon[29803]: pgmap v496: 196 pgs: 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:06:01.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:01 smithi111 ceph-mon[40056]: osdmap e361: 8 total, 8 up, 8 in 2024-09-16T09:06:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:01 smithi111 ceph-mon[40056]: pgmap v496: 196 pgs: 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:06:01.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:02.163 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:06:01 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:06:02.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:02 smithi114 ceph-mon[33315]: osdmap e362: 8 total, 8 up, 8 in 2024-09-16T09:06:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-57","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-57","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:02 smithi111 ceph-mon[29803]: osdmap e362: 8 total, 8 up, 8 in 2024-09-16T09:06:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-57","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-57","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:02 smithi111 ceph-mon[40056]: osdmap e362: 8 total, 8 up, 8 in 2024-09-16T09:06:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-57","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-57","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:03.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:03 smithi114 ceph-mon[33315]: osdmap e363: 8 total, 8 up, 8 in 2024-09-16T09:06:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-57"} : dispatch 2024-09-16T09:06:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-57"} : dispatch 2024-09-16T09:06:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:03 smithi114 ceph-mon[33315]: pgmap v499: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:06:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:06:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:06:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[29803]: osdmap e363: 8 total, 8 up, 8 in 2024-09-16T09:06:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-57"} : dispatch 2024-09-16T09:06:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-57"} : dispatch 2024-09-16T09:06:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[29803]: pgmap v499: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:06:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[40056]: osdmap e363: 8 total, 8 up, 8 in 2024-09-16T09:06:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-57"} : dispatch 2024-09-16T09:06:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-57"} : dispatch 2024-09-16T09:06:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:03 smithi111 ceph-mon[40056]: pgmap v499: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:06:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:04 smithi114 ceph-mon[33315]: osdmap e364: 8 total, 8 up, 8 in 2024-09-16T09:06:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:04 smithi111 ceph-mon[29803]: osdmap e364: 8 total, 8 up, 8 in 2024-09-16T09:06:04.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:04 smithi111 ceph-mon[40056]: osdmap e364: 8 total, 8 up, 8 in 2024-09-16T09:06:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:05 smithi114 ceph-mon[33315]: osdmap e365: 8 total, 8 up, 8 in 2024-09-16T09:06:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-59","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-59","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:05 smithi114 ceph-mon[33315]: pgmap v502: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:06:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:05 smithi111 ceph-mon[29803]: osdmap e365: 8 total, 8 up, 8 in 2024-09-16T09:06:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-59","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-59","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:05 smithi111 ceph-mon[29803]: pgmap v502: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:06:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:05 smithi111 ceph-mon[40056]: osdmap e365: 8 total, 8 up, 8 in 2024-09-16T09:06:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-59","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-59","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:05 smithi111 ceph-mon[40056]: pgmap v502: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-16T09:06:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:06:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[29803]: osdmap e366: 8 total, 8 up, 8 in 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-59"} : dispatch 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-59"} : dispatch 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[40056]: osdmap e366: 8 total, 8 up, 8 in 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-59"} : dispatch 2024-09-16T09:06:06.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-59"} : dispatch 2024-09-16T09:06:06.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:06 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:06:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:06 smithi114 ceph-mon[33315]: osdmap e366: 8 total, 8 up, 8 in 2024-09-16T09:06:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-59"} : dispatch 2024-09-16T09:06:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-59"} : dispatch 2024-09-16T09:06:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:07 smithi111 ceph-mon[29803]: osdmap e367: 8 total, 8 up, 8 in 2024-09-16T09:06:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:07 smithi111 ceph-mon[29803]: pgmap v505: 196 pgs: 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-16T09:06:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:07 smithi111 ceph-mon[40056]: osdmap e367: 8 total, 8 up, 8 in 2024-09-16T09:06:07.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:07 smithi111 ceph-mon[40056]: pgmap v505: 196 pgs: 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-16T09:06:07.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:07 smithi114 ceph-mon[33315]: osdmap e367: 8 total, 8 up, 8 in 2024-09-16T09:06:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:07 smithi114 ceph-mon[33315]: pgmap v505: 196 pgs: 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-16T09:06:08.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:08 smithi114 ceph-mon[33315]: osdmap e368: 8 total, 8 up, 8 in 2024-09-16T09:06:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-61","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-61","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:08 smithi111 ceph-mon[29803]: osdmap e368: 8 total, 8 up, 8 in 2024-09-16T09:06:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-61","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-61","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:08.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:08 smithi111 ceph-mon[40056]: osdmap e368: 8 total, 8 up, 8 in 2024-09-16T09:06:08.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-61","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-61","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:09 smithi114 ceph-mon[33315]: osdmap e369: 8 total, 8 up, 8 in 2024-09-16T09:06:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:09 smithi114 ceph-mon[33315]: pgmap v508: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-16T09:06:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[29803]: osdmap e369: 8 total, 8 up, 8 in 2024-09-16T09:06:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[29803]: pgmap v508: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-16T09:06:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[40056]: osdmap e369: 8 total, 8 up, 8 in 2024-09-16T09:06:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:09 smithi111 ceph-mon[40056]: pgmap v508: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-16T09:06:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:06:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:10 smithi114 ceph-mon[33315]: osdmap e370: 8 total, 8 up, 8 in 2024-09-16T09:06:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-61"} : dispatch 2024-09-16T09:06:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-61"} : dispatch 2024-09-16T09:06:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:06:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[29803]: osdmap e370: 8 total, 8 up, 8 in 2024-09-16T09:06:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-61"} : dispatch 2024-09-16T09:06:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-61"} : dispatch 2024-09-16T09:06:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:06:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[40056]: osdmap e370: 8 total, 8 up, 8 in 2024-09-16T09:06:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-61"} : dispatch 2024-09-16T09:06:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-61"} : dispatch 2024-09-16T09:06:11.831 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:11 smithi114 ceph-mon[33315]: osdmap e371: 8 total, 8 up, 8 in 2024-09-16T09:06:11.831 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:11 smithi114 ceph-mon[33315]: pgmap v511: 196 pgs: 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:11.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:11 smithi114 ceph-mon[33315]: osdmap e372: 8 total, 8 up, 8 in 2024-09-16T09:06:11.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-63","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:11.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-63","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[29803]: osdmap e371: 8 total, 8 up, 8 in 2024-09-16T09:06:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[29803]: pgmap v511: 196 pgs: 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[29803]: osdmap e372: 8 total, 8 up, 8 in 2024-09-16T09:06:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-63","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:11.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-63","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[40056]: osdmap e371: 8 total, 8 up, 8 in 2024-09-16T09:06:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[40056]: pgmap v511: 196 pgs: 196 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[40056]: osdmap e372: 8 total, 8 up, 8 in 2024-09-16T09:06:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-63","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-63","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:12.163 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:06:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:06:12.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:12 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:12 smithi114 ceph-mon[33315]: osdmap e373: 8 total, 8 up, 8 in 2024-09-16T09:06:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:12.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[29803]: osdmap e373: 8 total, 8 up, 8 in 2024-09-16T09:06:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:12.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[40056]: osdmap e373: 8 total, 8 up, 8 in 2024-09-16T09:06:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:13.478 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:06:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:06:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:06:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:13 smithi114 ceph-mon[33315]: pgmap v514: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:06:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:13 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:06:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:06:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:13 smithi114 ceph-mon[33315]: osdmap e374: 8 total, 8 up, 8 in 2024-09-16T09:06:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:13 smithi111 ceph-mon[29803]: pgmap v514: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:06:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:13 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:06:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:06:13.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:13 smithi111 ceph-mon[29803]: osdmap e374: 8 total, 8 up, 8 in 2024-09-16T09:06:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:13 smithi111 ceph-mon[40056]: pgmap v514: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:06:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:13 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:06:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:06:13.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:13 smithi111 ceph-mon[40056]: osdmap e374: 8 total, 8 up, 8 in 2024-09-16T09:06:15.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:15 smithi114 ceph-mon[33315]: osdmap e375: 8 total, 8 up, 8 in 2024-09-16T09:06:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:15 smithi114 ceph-mon[33315]: pgmap v517: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:06:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:15 smithi111 ceph-mon[40056]: osdmap e375: 8 total, 8 up, 8 in 2024-09-16T09:06:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:15 smithi111 ceph-mon[40056]: pgmap v517: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:06:15.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:15 smithi111 ceph-mon[29803]: osdmap e375: 8 total, 8 up, 8 in 2024-09-16T09:06:15.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:15 smithi111 ceph-mon[29803]: pgmap v517: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:06:16.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:16 smithi114 ceph-mon[33315]: osdmap e376: 8 total, 8 up, 8 in 2024-09-16T09:06:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:16.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:16 smithi111 ceph-mon[29803]: osdmap e376: 8 total, 8 up, 8 in 2024-09-16T09:06:16.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:16.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:16 smithi111 ceph-mon[40056]: osdmap e376: 8 total, 8 up, 8 in 2024-09-16T09:06:16.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:17.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:17 smithi114 ceph-mon[33315]: osdmap e377: 8 total, 8 up, 8 in 2024-09-16T09:06:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:17 smithi114 ceph-mon[33315]: pgmap v520: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-16T09:06:17.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:17 smithi111 ceph-mon[40056]: osdmap e377: 8 total, 8 up, 8 in 2024-09-16T09:06:17.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:17 smithi111 ceph-mon[40056]: pgmap v520: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-16T09:06:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:17 smithi111 ceph-mon[29803]: osdmap e377: 8 total, 8 up, 8 in 2024-09-16T09:06:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:17 smithi111 ceph-mon[29803]: pgmap v520: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-16T09:06:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:18 smithi114 ceph-mon[33315]: osdmap e378: 8 total, 8 up, 8 in 2024-09-16T09:06:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:18 smithi111 ceph-mon[29803]: osdmap e378: 8 total, 8 up, 8 in 2024-09-16T09:06:18.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:18 smithi111 ceph-mon[40056]: osdmap e378: 8 total, 8 up, 8 in 2024-09-16T09:06:19.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:19 smithi114 ceph-mon[33315]: pgmap v522: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-16T09:06:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:19 smithi111 ceph-mon[29803]: pgmap v522: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-16T09:06:19.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:19 smithi111 ceph-mon[40056]: pgmap v522: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-16T09:06:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:21 smithi114 ceph-mon[33315]: pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-09-16T09:06:21.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:06:21 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:06:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:21 smithi111 ceph-mon[29803]: pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-09-16T09:06:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:21 smithi111 ceph-mon[40056]: pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-09-16T09:06:22.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:23.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:06:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:06:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:06:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:24 smithi114 ceph-mon[33315]: pgmap v524: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-09-16T09:06:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:24 smithi114 ceph-mon[33315]: osdmap e379: 8 total, 8 up, 8 in 2024-09-16T09:06:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:24 smithi111 ceph-mon[29803]: pgmap v524: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-09-16T09:06:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:24 smithi111 ceph-mon[29803]: osdmap e379: 8 total, 8 up, 8 in 2024-09-16T09:06:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:24 smithi111 ceph-mon[40056]: pgmap v524: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-09-16T09:06:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:24 smithi111 ceph-mon[40056]: osdmap e379: 8 total, 8 up, 8 in 2024-09-16T09:06:26.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:26 smithi114 ceph-mon[33315]: pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 0 op/s 2024-09-16T09:06:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:26 smithi111 ceph-mon[29803]: pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 0 op/s 2024-09-16T09:06:26.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:26 smithi111 ceph-mon[40056]: pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 0 op/s 2024-09-16T09:06:28.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:28 smithi114 ceph-mon[33315]: pgmap v527: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:28 smithi111 ceph-mon[29803]: pgmap v527: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:28.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:28 smithi111 ceph-mon[40056]: pgmap v527: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:29.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:29 smithi114 ceph-mon[33315]: osdmap e380: 8 total, 8 up, 8 in 2024-09-16T09:06:29.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:29 smithi111 ceph-mon[29803]: osdmap e380: 8 total, 8 up, 8 in 2024-09-16T09:06:29.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:29 smithi111 ceph-mon[40056]: osdmap e380: 8 total, 8 up, 8 in 2024-09-16T09:06:30.577 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:30 smithi111 ceph-mon[40056]: pgmap v529: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:30.578 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:30 smithi111 ceph-mon[29803]: pgmap v529: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:30.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:30 smithi114 ceph-mon[33315]: pgmap v529: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:31.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:32.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:06:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:06:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:32 smithi114 ceph-mon[33315]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:06:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:32 smithi111 ceph-mon[29803]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:06:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:32 smithi111 ceph-mon[40056]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:06:33.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:33 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:33 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:06:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:06:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:06:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:33 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:34 smithi114 ceph-mon[33315]: pgmap v531: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:34 smithi111 ceph-mon[29803]: pgmap v531: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:34.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:34 smithi111 ceph-mon[40056]: pgmap v531: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:36 smithi114 ceph-mon[33315]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:36 smithi111 ceph-mon[29803]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:36.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:36 smithi111 ceph-mon[40056]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:38.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:38 smithi114 ceph-mon[33315]: pgmap v533: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:38 smithi114 ceph-mon[33315]: osdmap e381: 8 total, 8 up, 8 in 2024-09-16T09:06:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:38 smithi111 ceph-mon[29803]: pgmap v533: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:38 smithi111 ceph-mon[29803]: osdmap e381: 8 total, 8 up, 8 in 2024-09-16T09:06:38.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:38 smithi111 ceph-mon[40056]: pgmap v533: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:38.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:38 smithi111 ceph-mon[40056]: osdmap e381: 8 total, 8 up, 8 in 2024-09-16T09:06:40.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:40 smithi114 ceph-mon[33315]: pgmap v535: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:40 smithi114 ceph-mon[33315]: osdmap e382: 8 total, 8 up, 8 in 2024-09-16T09:06:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:40 smithi111 ceph-mon[29803]: pgmap v535: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:40 smithi111 ceph-mon[29803]: osdmap e382: 8 total, 8 up, 8 in 2024-09-16T09:06:40.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:40 smithi111 ceph-mon[40056]: pgmap v535: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:06:40.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:40 smithi111 ceph-mon[40056]: osdmap e382: 8 total, 8 up, 8 in 2024-09-16T09:06:42.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:06:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:06:42.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:42 smithi114 ceph-mon[33315]: pgmap v537: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:06:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:42 smithi111 ceph-mon[29803]: pgmap v537: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:06:42.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:42 smithi111 ceph-mon[40056]: pgmap v537: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:06:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:43 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:43 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:43.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:43 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:06:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:06:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:06:44.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:44 smithi114 ceph-mon[33315]: pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-16T09:06:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:44 smithi111 ceph-mon[29803]: pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-16T09:06:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:44 smithi111 ceph-mon[40056]: pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-16T09:06:46.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:46 smithi114 ceph-mon[33315]: pgmap v539: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-16T09:06:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:46 smithi111 ceph-mon[29803]: pgmap v539: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-16T09:06:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:46.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:46 smithi111 ceph-mon[40056]: pgmap v539: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-16T09:06:46.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:06:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:06:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:06:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:06:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:06:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:06:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:06:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:06:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:06:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:06:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:06:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:06:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:06:48.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:48 smithi114 ceph-mon[33315]: pgmap v540: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-16T09:06:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:48 smithi114 ceph-mon[33315]: osdmap e383: 8 total, 8 up, 8 in 2024-09-16T09:06:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:48 smithi111 ceph-mon[29803]: pgmap v540: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-16T09:06:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:48 smithi111 ceph-mon[29803]: osdmap e383: 8 total, 8 up, 8 in 2024-09-16T09:06:48.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:48 smithi111 ceph-mon[40056]: pgmap v540: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-16T09:06:48.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:48 smithi111 ceph-mon[40056]: osdmap e383: 8 total, 8 up, 8 in 2024-09-16T09:06:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:50 smithi114 ceph-mon[33315]: pgmap v542: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:06:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-63"} : dispatch 2024-09-16T09:06:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-63"} : dispatch 2024-09-16T09:06:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[29803]: pgmap v542: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:06:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-63"} : dispatch 2024-09-16T09:06:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-63"} : dispatch 2024-09-16T09:06:50.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[40056]: pgmap v542: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:06:50.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:06:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-63"} : dispatch 2024-09-16T09:06:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-63"} : dispatch 2024-09-16T09:06:51.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:51 smithi114 ceph-mon[33315]: osdmap e384: 8 total, 8 up, 8 in 2024-09-16T09:06:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:51 smithi111 ceph-mon[29803]: osdmap e384: 8 total, 8 up, 8 in 2024-09-16T09:06:51.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:51 smithi111 ceph-mon[40056]: osdmap e384: 8 total, 8 up, 8 in 2024-09-16T09:06:52.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:06:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:06:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:52 smithi114 ceph-mon[33315]: pgmap v544: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:52 smithi114 ceph-mon[33315]: osdmap e385: 8 total, 8 up, 8 in 2024-09-16T09:06:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-65","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-65","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:52 smithi111 ceph-mon[29803]: pgmap v544: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:52 smithi111 ceph-mon[29803]: osdmap e385: 8 total, 8 up, 8 in 2024-09-16T09:06:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-65","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:52.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-65","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:52.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:52 smithi111 ceph-mon[40056]: pgmap v544: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:52 smithi111 ceph-mon[40056]: osdmap e385: 8 total, 8 up, 8 in 2024-09-16T09:06:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-65","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:52.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-65","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:06:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:53 smithi114 ceph-mon[33315]: osdmap e386: 8 total, 8 up, 8 in 2024-09-16T09:06:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:06:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:06:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[29803]: osdmap e386: 8 total, 8 up, 8 in 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[40056]: osdmap e386: 8 total, 8 up, 8 in 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:06:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:53.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:06:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:54 smithi114 ceph-mon[33315]: pgmap v547: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:54.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:54 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:06:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:54 smithi114 ceph-mon[33315]: osdmap e387: 8 total, 8 up, 8 in 2024-09-16T09:06:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:54 smithi111 ceph-mon[29803]: pgmap v547: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:54 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:06:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:54 smithi111 ceph-mon[29803]: osdmap e387: 8 total, 8 up, 8 in 2024-09-16T09:06:54.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:54 smithi111 ceph-mon[40056]: pgmap v547: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:06:54.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:54 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:06:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:54 smithi111 ceph-mon[40056]: osdmap e387: 8 total, 8 up, 8 in 2024-09-16T09:06:55.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:55 smithi114 ceph-mon[33315]: osdmap e388: 8 total, 8 up, 8 in 2024-09-16T09:06:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:55 smithi111 ceph-mon[29803]: osdmap e388: 8 total, 8 up, 8 in 2024-09-16T09:06:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:55 smithi111 ceph-mon[40056]: osdmap e388: 8 total, 8 up, 8 in 2024-09-16T09:06:56.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:56 smithi114 ceph-mon[33315]: pgmap v550: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:06:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:56 smithi114 ceph-mon[33315]: osdmap e389: 8 total, 8 up, 8 in 2024-09-16T09:06:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:56 smithi111 ceph-mon[29803]: pgmap v550: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:06:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:56 smithi111 ceph-mon[29803]: osdmap e389: 8 total, 8 up, 8 in 2024-09-16T09:06:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:56 smithi111 ceph-mon[40056]: pgmap v550: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:06:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:56 smithi111 ceph-mon[40056]: osdmap e389: 8 total, 8 up, 8 in 2024-09-16T09:06:57.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:57 smithi114 ceph-mon[33315]: osdmap e390: 8 total, 8 up, 8 in 2024-09-16T09:06:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:57 smithi114 ceph-mon[33315]: pgmap v553: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-09-16T09:06:57.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:57 smithi111 ceph-mon[40056]: osdmap e390: 8 total, 8 up, 8 in 2024-09-16T09:06:57.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:57 smithi111 ceph-mon[40056]: pgmap v553: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-09-16T09:06:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:57 smithi111 ceph-mon[29803]: osdmap e390: 8 total, 8 up, 8 in 2024-09-16T09:06:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:57 smithi111 ceph-mon[29803]: pgmap v553: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-09-16T09:06:59.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:06:59 smithi114 ceph-mon[33315]: pgmap v554: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-16T09:06:59.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:06:59 smithi111 ceph-mon[29803]: pgmap v554: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-16T09:06:59.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:06:59 smithi111 ceph-mon[40056]: pgmap v554: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-16T09:07:01.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:01 smithi114 ceph-mon[33315]: pgmap v555: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-09-16T09:07:01.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:01.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:07:01 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:07:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:01 smithi111 ceph-mon[29803]: pgmap v555: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-09-16T09:07:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:01.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:01 smithi111 ceph-mon[40056]: pgmap v555: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-09-16T09:07:01.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:02.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:02.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:02.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:07:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:07:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:07:04.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:04 smithi114 ceph-mon[33315]: pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-16T09:07:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:04 smithi114 ceph-mon[33315]: osdmap e391: 8 total, 8 up, 8 in 2024-09-16T09:07:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:04 smithi111 ceph-mon[29803]: pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-16T09:07:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:04 smithi111 ceph-mon[29803]: osdmap e391: 8 total, 8 up, 8 in 2024-09-16T09:07:04.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:04 smithi111 ceph-mon[40056]: pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-16T09:07:04.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:04 smithi111 ceph-mon[40056]: osdmap e391: 8 total, 8 up, 8 in 2024-09-16T09:07:06.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:06 smithi114 ceph-mon[33315]: pgmap v558: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-16T09:07:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:06 smithi111 ceph-mon[29803]: pgmap v558: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-16T09:07:06.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:06 smithi111 ceph-mon[40056]: pgmap v558: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-16T09:07:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-65"} : dispatch 2024-09-16T09:07:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-65"} : dispatch 2024-09-16T09:07:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-65"} : dispatch 2024-09-16T09:07:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-65"} : dispatch 2024-09-16T09:07:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:07.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-65"} : dispatch 2024-09-16T09:07:07.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-65"} : dispatch 2024-09-16T09:07:08.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:08 smithi114 ceph-mon[33315]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:07:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:08 smithi114 ceph-mon[33315]: osdmap e392: 8 total, 8 up, 8 in 2024-09-16T09:07:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:08 smithi111 ceph-mon[29803]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:07:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:08 smithi111 ceph-mon[29803]: osdmap e392: 8 total, 8 up, 8 in 2024-09-16T09:07:08.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:08 smithi111 ceph-mon[40056]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:07:08.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:08 smithi111 ceph-mon[40056]: osdmap e392: 8 total, 8 up, 8 in 2024-09-16T09:07:09.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:09 smithi114 ceph-mon[33315]: osdmap e393: 8 total, 8 up, 8 in 2024-09-16T09:07:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-67","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-67","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:09.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:09 smithi111 ceph-mon[29803]: osdmap e393: 8 total, 8 up, 8 in 2024-09-16T09:07:09.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-67","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:09.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-67","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:09 smithi111 ceph-mon[40056]: osdmap e393: 8 total, 8 up, 8 in 2024-09-16T09:07:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-67","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:09.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-67","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:10 smithi114 ceph-mon[33315]: pgmap v562: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:10 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:10 smithi114 ceph-mon[33315]: osdmap e394: 8 total, 8 up, 8 in 2024-09-16T09:07:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[29803]: pgmap v562: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[29803]: osdmap e394: 8 total, 8 up, 8 in 2024-09-16T09:07:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[40056]: pgmap v562: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[40056]: osdmap e394: 8 total, 8 up, 8 in 2024-09-16T09:07:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:11.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:11 smithi114 ceph-mon[33315]: osdmap e395: 8 total, 8 up, 8 in 2024-09-16T09:07:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-67"} : dispatch 2024-09-16T09:07:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-67"} : dispatch 2024-09-16T09:07:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[29803]: osdmap e395: 8 total, 8 up, 8 in 2024-09-16T09:07:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-67"} : dispatch 2024-09-16T09:07:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-67"} : dispatch 2024-09-16T09:07:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[40056]: osdmap e395: 8 total, 8 up, 8 in 2024-09-16T09:07:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-67"} : dispatch 2024-09-16T09:07:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-67"} : dispatch 2024-09-16T09:07:12.163 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:07:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:07:12.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:12 smithi114 ceph-mon[33315]: pgmap v565: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:12 smithi114 ceph-mon[33315]: osdmap e396: 8 total, 8 up, 8 in 2024-09-16T09:07:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:12 smithi111 ceph-mon[29803]: pgmap v565: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:12 smithi111 ceph-mon[29803]: osdmap e396: 8 total, 8 up, 8 in 2024-09-16T09:07:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:12 smithi111 ceph-mon[40056]: pgmap v565: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:12 smithi111 ceph-mon[40056]: osdmap e396: 8 total, 8 up, 8 in 2024-09-16T09:07:13.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:13 smithi114 ceph-mon[33315]: osdmap e397: 8 total, 8 up, 8 in 2024-09-16T09:07:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-69","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-69","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:13 smithi111 ceph-mon[29803]: osdmap e397: 8 total, 8 up, 8 in 2024-09-16T09:07:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-69","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-69","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:13.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:07:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:07:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:07:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:13 smithi111 ceph-mon[40056]: osdmap e397: 8 total, 8 up, 8 in 2024-09-16T09:07:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-69","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-69","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:14.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:14 smithi114 ceph-mon[33315]: pgmap v568: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:07:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:14 smithi114 ceph-mon[33315]: osdmap e398: 8 total, 8 up, 8 in 2024-09-16T09:07:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-69"} : dispatch 2024-09-16T09:07:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-69"} : dispatch 2024-09-16T09:07:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[29803]: pgmap v568: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:07:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[29803]: osdmap e398: 8 total, 8 up, 8 in 2024-09-16T09:07:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-69"} : dispatch 2024-09-16T09:07:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-69"} : dispatch 2024-09-16T09:07:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[40056]: pgmap v568: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:07:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[40056]: osdmap e398: 8 total, 8 up, 8 in 2024-09-16T09:07:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:14.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-69"} : dispatch 2024-09-16T09:07:14.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-69"} : dispatch 2024-09-16T09:07:15.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:15 smithi114 ceph-mon[33315]: osdmap e399: 8 total, 8 up, 8 in 2024-09-16T09:07:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:15 smithi111 ceph-mon[29803]: osdmap e399: 8 total, 8 up, 8 in 2024-09-16T09:07:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:15 smithi111 ceph-mon[40056]: osdmap e399: 8 total, 8 up, 8 in 2024-09-16T09:07:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:16 smithi114 ceph-mon[33315]: pgmap v571: 196 pgs: 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:07:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:16 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:16 smithi114 ceph-mon[33315]: osdmap e400: 8 total, 8 up, 8 in 2024-09-16T09:07:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-71","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-71","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[29803]: pgmap v571: 196 pgs: 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:07:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[29803]: osdmap e400: 8 total, 8 up, 8 in 2024-09-16T09:07:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-71","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-71","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[40056]: pgmap v571: 196 pgs: 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:07:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[40056]: osdmap e400: 8 total, 8 up, 8 in 2024-09-16T09:07:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-71","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-71","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:17.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:17 smithi114 ceph-mon[33315]: osdmap e401: 8 total, 8 up, 8 in 2024-09-16T09:07:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:07:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:07:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:17 smithi114 ceph-mon[33315]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:07:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:07:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:17 smithi114 ceph-mon[33315]: osdmap e402: 8 total, 8 up, 8 in 2024-09-16T09:07:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[29803]: osdmap e401: 8 total, 8 up, 8 in 2024-09-16T09:07:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:07:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:07:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[29803]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:07:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:07:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[29803]: osdmap e402: 8 total, 8 up, 8 in 2024-09-16T09:07:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[40056]: osdmap e401: 8 total, 8 up, 8 in 2024-09-16T09:07:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:07:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:07:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[40056]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:07:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:07:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:17 smithi111 ceph-mon[40056]: osdmap e402: 8 total, 8 up, 8 in 2024-09-16T09:07:19.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:19 smithi114 ceph-mon[33315]: osdmap e403: 8 total, 8 up, 8 in 2024-09-16T09:07:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:19 smithi114 ceph-mon[33315]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:07:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:19 smithi111 ceph-mon[29803]: osdmap e403: 8 total, 8 up, 8 in 2024-09-16T09:07:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:19 smithi111 ceph-mon[29803]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:07:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:19 smithi111 ceph-mon[40056]: osdmap e403: 8 total, 8 up, 8 in 2024-09-16T09:07:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:19 smithi111 ceph-mon[40056]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:07:21.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:21 smithi114 ceph-mon[33315]: pgmap v578: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 586 B/s wr, 1 op/s 2024-09-16T09:07:21.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:07:21 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:07:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:21 smithi111 ceph-mon[29803]: pgmap v578: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 586 B/s wr, 1 op/s 2024-09-16T09:07:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:21 smithi111 ceph-mon[40056]: pgmap v578: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 586 B/s wr, 1 op/s 2024-09-16T09:07:22.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:23.659 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:07:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:07:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:07:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:23 smithi114 ceph-mon[33315]: pgmap v579: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-09-16T09:07:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:23 smithi111 ceph-mon[29803]: pgmap v579: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-09-16T09:07:23.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:23 smithi111 ceph-mon[40056]: pgmap v579: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-09-16T09:07:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:25 smithi114 ceph-mon[33315]: pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-16T09:07:25.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:25 smithi111 ceph-mon[40056]: pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-16T09:07:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:25 smithi111 ceph-mon[29803]: pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-16T09:07:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:27 smithi111 ceph-mon[29803]: pgmap v581: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-09-16T09:07:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:27 smithi111 ceph-mon[40056]: pgmap v581: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-09-16T09:07:28.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:27 smithi114 ceph-mon[33315]: pgmap v581: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-09-16T09:07:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:29 smithi111 ceph-mon[29803]: pgmap v582: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-09-16T09:07:29.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:29 smithi111 ceph-mon[40056]: pgmap v582: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-09-16T09:07:30.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:29 smithi114 ceph-mon[33315]: pgmap v582: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-09-16T09:07:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:31 smithi114 ceph-mon[33315]: pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-16T09:07:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.19", "id": [7, 3]} : dispatch 2024-09-16T09:07:32.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:07:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:07:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:31 smithi111 ceph-mon[29803]: pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-16T09:07:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.19", "id": [7, 3]} : dispatch 2024-09-16T09:07:32.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:31 smithi111 ceph-mon[40056]: pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-16T09:07:32.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.19", "id": [7, 3]} : dispatch 2024-09-16T09:07:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.19", "id": [7, 3]}]': finished 2024-09-16T09:07:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:32 smithi114 ceph-mon[33315]: osdmap e404: 8 total, 8 up, 8 in 2024-09-16T09:07:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.19", "id": [7, 3]}]': finished 2024-09-16T09:07:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:32 smithi111 ceph-mon[29803]: osdmap e404: 8 total, 8 up, 8 in 2024-09-16T09:07:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:33.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.19", "id": [7, 3]}]': finished 2024-09-16T09:07:33.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:32 smithi111 ceph-mon[40056]: osdmap e404: 8 total, 8 up, 8 in 2024-09-16T09:07:33.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:07:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:07:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:07:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:33 smithi114 ceph-mon[33315]: pgmap v585: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-09-16T09:07:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:33 smithi114 ceph-mon[33315]: osdmap e405: 8 total, 8 up, 8 in 2024-09-16T09:07:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:33 smithi111 ceph-mon[29803]: pgmap v585: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-09-16T09:07:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:33 smithi111 ceph-mon[29803]: osdmap e405: 8 total, 8 up, 8 in 2024-09-16T09:07:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:33 smithi111 ceph-mon[40056]: pgmap v585: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-09-16T09:07:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:33 smithi111 ceph-mon[40056]: osdmap e405: 8 total, 8 up, 8 in 2024-09-16T09:07:36.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:35 smithi114 ceph-mon[33315]: pgmap v587: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 383 B/s wr, 2 op/s 2024-09-16T09:07:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:35 smithi111 ceph-mon[29803]: pgmap v587: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 383 B/s wr, 2 op/s 2024-09-16T09:07:36.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:35 smithi111 ceph-mon[40056]: pgmap v587: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 383 B/s wr, 2 op/s 2024-09-16T09:07:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:37 smithi114 ceph-mon[33315]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-09-16T09:07:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:37 smithi111 ceph-mon[29803]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-09-16T09:07:38.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:37 smithi111 ceph-mon[40056]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-09-16T09:07:39.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-71"} : dispatch 2024-09-16T09:07:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-71"} : dispatch 2024-09-16T09:07:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-71"} : dispatch 2024-09-16T09:07:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-71"} : dispatch 2024-09-16T09:07:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-71"} : dispatch 2024-09-16T09:07:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-71"} : dispatch 2024-09-16T09:07:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:39 smithi114 ceph-mon[33315]: pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-09-16T09:07:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:39 smithi114 ceph-mon[33315]: osdmap e406: 8 total, 8 up, 8 in 2024-09-16T09:07:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:39 smithi111 ceph-mon[29803]: pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-09-16T09:07:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:39 smithi111 ceph-mon[29803]: osdmap e406: 8 total, 8 up, 8 in 2024-09-16T09:07:40.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:39 smithi111 ceph-mon[40056]: pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-09-16T09:07:40.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:39 smithi111 ceph-mon[40056]: osdmap e406: 8 total, 8 up, 8 in 2024-09-16T09:07:41.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:40 smithi114 ceph-mon[33315]: osdmap e407: 8 total, 8 up, 8 in 2024-09-16T09:07:41.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-73","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:41.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-73","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:41.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:40 smithi111 ceph-mon[29803]: osdmap e407: 8 total, 8 up, 8 in 2024-09-16T09:07:41.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-73","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:41.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-73","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:41.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:40 smithi111 ceph-mon[40056]: osdmap e407: 8 total, 8 up, 8 in 2024-09-16T09:07:41.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-73","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:41.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-73","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:42.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:07:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:07:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:41 smithi114 ceph-mon[33315]: pgmap v592: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:41 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:41 smithi114 ceph-mon[33315]: osdmap e408: 8 total, 8 up, 8 in 2024-09-16T09:07:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[29803]: pgmap v592: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[29803]: osdmap e408: 8 total, 8 up, 8 in 2024-09-16T09:07:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[40056]: pgmap v592: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[40056]: osdmap e408: 8 total, 8 up, 8 in 2024-09-16T09:07:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:42 smithi114 ceph-mon[33315]: osdmap e409: 8 total, 8 up, 8 in 2024-09-16T09:07:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:42 smithi111 ceph-mon[29803]: osdmap e409: 8 total, 8 up, 8 in 2024-09-16T09:07:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:42 smithi111 ceph-mon[40056]: osdmap e409: 8 total, 8 up, 8 in 2024-09-16T09:07:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:07:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:07:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:07:44.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:43 smithi114 ceph-mon[33315]: pgmap v595: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:43 smithi114 ceph-mon[33315]: osdmap e410: 8 total, 8 up, 8 in 2024-09-16T09:07:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:43 smithi111 ceph-mon[29803]: pgmap v595: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:43 smithi111 ceph-mon[29803]: osdmap e410: 8 total, 8 up, 8 in 2024-09-16T09:07:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:43 smithi111 ceph-mon[40056]: pgmap v595: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:07:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:43 smithi111 ceph-mon[40056]: osdmap e410: 8 total, 8 up, 8 in 2024-09-16T09:07:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:45 smithi114 ceph-mon[33315]: pgmap v597: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-09-16T09:07:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:45 smithi111 ceph-mon[29803]: pgmap v597: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-09-16T09:07:46.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:45 smithi111 ceph-mon[40056]: pgmap v597: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-09-16T09:07:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:07:48.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:47 smithi114 ceph-mon[33315]: pgmap v598: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-16T09:07:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:07:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:07:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:07:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:07:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[29803]: pgmap v598: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-16T09:07:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:07:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:07:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:07:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:07:48.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[40056]: pgmap v598: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-16T09:07:48.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:07:48.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:07:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:07:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:07:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:48 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:48 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:48 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:07:50.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:49 smithi114 ceph-mon[33315]: pgmap v599: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 908 B/s wr, 4 op/s 2024-09-16T09:07:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:49 smithi111 ceph-mon[29803]: pgmap v599: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 908 B/s wr, 4 op/s 2024-09-16T09:07:50.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:49 smithi111 ceph-mon[40056]: pgmap v599: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 908 B/s wr, 4 op/s 2024-09-16T09:07:52.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:51 smithi114 ceph-mon[33315]: pgmap v600: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 230 B/s wr, 2 op/s 2024-09-16T09:07:52.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:07:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:07:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:51 smithi111 ceph-mon[29803]: pgmap v600: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 230 B/s wr, 2 op/s 2024-09-16T09:07:52.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:51 smithi111 ceph-mon[40056]: pgmap v600: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 230 B/s wr, 2 op/s 2024-09-16T09:07:53.064 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:52 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:53.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:52 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:52 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:07:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:07:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:07:54.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:53 smithi114 ceph-mon[33315]: pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 204 B/s wr, 2 op/s 2024-09-16T09:07:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-73"} : dispatch 2024-09-16T09:07:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-73"} : dispatch 2024-09-16T09:07:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[29803]: pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 204 B/s wr, 2 op/s 2024-09-16T09:07:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-73"} : dispatch 2024-09-16T09:07:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-73"} : dispatch 2024-09-16T09:07:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[40056]: pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 204 B/s wr, 2 op/s 2024-09-16T09:07:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-73"} : dispatch 2024-09-16T09:07:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-73"} : dispatch 2024-09-16T09:07:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:54 smithi114 ceph-mon[33315]: osdmap e411: 8 total, 8 up, 8 in 2024-09-16T09:07:55.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:54 smithi111 ceph-mon[29803]: osdmap e411: 8 total, 8 up, 8 in 2024-09-16T09:07:55.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:54 smithi111 ceph-mon[40056]: osdmap e411: 8 total, 8 up, 8 in 2024-09-16T09:07:56.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:55 smithi114 ceph-mon[33315]: pgmap v603: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:07:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:55 smithi114 ceph-mon[33315]: osdmap e412: 8 total, 8 up, 8 in 2024-09-16T09:07:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-75","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-75","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:55 smithi111 ceph-mon[29803]: pgmap v603: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:07:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:55 smithi111 ceph-mon[29803]: osdmap e412: 8 total, 8 up, 8 in 2024-09-16T09:07:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-75","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-75","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:56.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:55 smithi111 ceph-mon[40056]: pgmap v603: 196 pgs: 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:07:56.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:55 smithi111 ceph-mon[40056]: osdmap e412: 8 total, 8 up, 8 in 2024-09-16T09:07:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-75","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-75","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:07:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:56 smithi114 ceph-mon[33315]: osdmap e413: 8 total, 8 up, 8 in 2024-09-16T09:07:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:56 smithi111 ceph-mon[29803]: osdmap e413: 8 total, 8 up, 8 in 2024-09-16T09:07:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:57.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:07:57.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:56 smithi111 ceph-mon[40056]: osdmap e413: 8 total, 8 up, 8 in 2024-09-16T09:07:57.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:07:58.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:57 smithi114 ceph-mon[33315]: pgmap v606: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:07:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:57 smithi114 ceph-mon[33315]: osdmap e414: 8 total, 8 up, 8 in 2024-09-16T09:07:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:57 smithi111 ceph-mon[29803]: pgmap v606: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:07:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:57 smithi111 ceph-mon[29803]: osdmap e414: 8 total, 8 up, 8 in 2024-09-16T09:07:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:57 smithi111 ceph-mon[40056]: pgmap v606: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:07:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:57 smithi111 ceph-mon[40056]: osdmap e414: 8 total, 8 up, 8 in 2024-09-16T09:07:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:58 smithi114 ceph-mon[33315]: osdmap e415: 8 total, 8 up, 8 in 2024-09-16T09:07:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-75"} : dispatch 2024-09-16T09:07:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-75"} : dispatch 2024-09-16T09:07:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[29803]: osdmap e415: 8 total, 8 up, 8 in 2024-09-16T09:07:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-75"} : dispatch 2024-09-16T09:07:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-75"} : dispatch 2024-09-16T09:07:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[40056]: osdmap e415: 8 total, 8 up, 8 in 2024-09-16T09:07:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:07:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-75"} : dispatch 2024-09-16T09:07:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-75"} : dispatch 2024-09-16T09:08:00.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:59 smithi114 ceph-mon[33315]: pgmap v609: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:08:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:07:59 smithi114 ceph-mon[33315]: osdmap e416: 8 total, 8 up, 8 in 2024-09-16T09:08:00.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:59 smithi111 ceph-mon[40056]: pgmap v609: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:08:00.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:07:59 smithi111 ceph-mon[40056]: osdmap e416: 8 total, 8 up, 8 in 2024-09-16T09:08:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:59 smithi111 ceph-mon[29803]: pgmap v609: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:08:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:07:59 smithi111 ceph-mon[29803]: osdmap e416: 8 total, 8 up, 8 in 2024-09-16T09:08:01.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:00 smithi114 ceph-mon[33315]: osdmap e417: 8 total, 8 up, 8 in 2024-09-16T09:08:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-77","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-77","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:01.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:00 smithi111 ceph-mon[29803]: osdmap e417: 8 total, 8 up, 8 in 2024-09-16T09:08:01.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-77","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:01.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-77","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:01.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:00 smithi111 ceph-mon[40056]: osdmap e417: 8 total, 8 up, 8 in 2024-09-16T09:08:01.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-77","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:01.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-77","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:02.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:08:01 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:08:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:01 smithi114 ceph-mon[33315]: pgmap v612: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:08:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:01 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:01 smithi114 ceph-mon[33315]: osdmap e418: 8 total, 8 up, 8 in 2024-09-16T09:08:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:02.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[40056]: pgmap v612: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:08:02.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:02.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:02.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[40056]: osdmap e418: 8 total, 8 up, 8 in 2024-09-16T09:08:02.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:02.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:02.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[29803]: pgmap v612: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:08:02.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:02.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:02.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[29803]: osdmap e418: 8 total, 8 up, 8 in 2024-09-16T09:08:02.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:02.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:02 smithi114 ceph-mon[33315]: osdmap e419: 8 total, 8 up, 8 in 2024-09-16T09:08:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-77"} : dispatch 2024-09-16T09:08:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-77"} : dispatch 2024-09-16T09:08:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[29803]: osdmap e419: 8 total, 8 up, 8 in 2024-09-16T09:08:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-77"} : dispatch 2024-09-16T09:08:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-77"} : dispatch 2024-09-16T09:08:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[40056]: osdmap e419: 8 total, 8 up, 8 in 2024-09-16T09:08:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-77"} : dispatch 2024-09-16T09:08:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-77"} : dispatch 2024-09-16T09:08:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:08:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:08:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:08:04.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:03 smithi114 ceph-mon[33315]: pgmap v615: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:08:04.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:03 smithi114 ceph-mon[33315]: osdmap e420: 8 total, 8 up, 8 in 2024-09-16T09:08:04.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:03 smithi111 ceph-mon[29803]: pgmap v615: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:08:04.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:03 smithi111 ceph-mon[29803]: osdmap e420: 8 total, 8 up, 8 in 2024-09-16T09:08:04.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:03 smithi111 ceph-mon[40056]: pgmap v615: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:08:04.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:03 smithi111 ceph-mon[40056]: osdmap e420: 8 total, 8 up, 8 in 2024-09-16T09:08:05.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:04 smithi114 ceph-mon[33315]: osdmap e421: 8 total, 8 up, 8 in 2024-09-16T09:08:05.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-79","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:05.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-79","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:05.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:04 smithi111 ceph-mon[40056]: osdmap e421: 8 total, 8 up, 8 in 2024-09-16T09:08:05.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-79","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:05.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-79","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:05.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:04 smithi111 ceph-mon[29803]: osdmap e421: 8 total, 8 up, 8 in 2024-09-16T09:08:05.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-79","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:05.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-79","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:05 smithi114 ceph-mon[33315]: pgmap v618: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:08:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:05 smithi114 ceph-mon[33315]: osdmap e422: 8 total, 8 up, 8 in 2024-09-16T09:08:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:05 smithi111 ceph-mon[29803]: pgmap v618: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:08:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:05 smithi111 ceph-mon[29803]: osdmap e422: 8 total, 8 up, 8 in 2024-09-16T09:08:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:06.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:05 smithi111 ceph-mon[40056]: pgmap v618: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:08:06.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:06.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:05 smithi111 ceph-mon[40056]: osdmap e422: 8 total, 8 up, 8 in 2024-09-16T09:08:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:06 smithi114 ceph-mon[33315]: osdmap e423: 8 total, 8 up, 8 in 2024-09-16T09:08:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:06 smithi111 ceph-mon[29803]: osdmap e423: 8 total, 8 up, 8 in 2024-09-16T09:08:07.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:06 smithi111 ceph-mon[40056]: osdmap e423: 8 total, 8 up, 8 in 2024-09-16T09:08:08.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:07 smithi114 ceph-mon[33315]: pgmap v621: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:08:08.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:07 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:08.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:07 smithi114 ceph-mon[33315]: osdmap e424: 8 total, 8 up, 8 in 2024-09-16T09:08:08.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "304.7"} : dispatch 2024-09-16T09:08:08.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:07 smithi114 ceph-mon[33315]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.7"}]: dispatch 2024-09-16T09:08:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[29803]: pgmap v621: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:08:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[29803]: osdmap e424: 8 total, 8 up, 8 in 2024-09-16T09:08:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "304.7"} : dispatch 2024-09-16T09:08:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[29803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.7"}]: dispatch 2024-09-16T09:08:08.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[40056]: pgmap v621: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:08:08.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:08.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[40056]: osdmap e424: 8 total, 8 up, 8 in 2024-09-16T09:08:08.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "304.7"} : dispatch 2024-09-16T09:08:08.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:07 smithi111 ceph-mon[40056]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.7"}]: dispatch 2024-09-16T09:08:09.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:08 smithi114 ceph-mon[33315]: 304.7 deep-scrub starts 2024-09-16T09:08:09.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:08 smithi114 ceph-mon[33315]: 304.7 deep-scrub ok 2024-09-16T09:08:09.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:08 smithi111 ceph-mon[29803]: 304.7 deep-scrub starts 2024-09-16T09:08:09.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:08 smithi111 ceph-mon[29803]: 304.7 deep-scrub ok 2024-09-16T09:08:09.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:08 smithi111 ceph-mon[40056]: 304.7 deep-scrub starts 2024-09-16T09:08:09.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:08 smithi111 ceph-mon[40056]: 304.7 deep-scrub ok 2024-09-16T09:08:10.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:09 smithi114 ceph-mon[33315]: pgmap v623: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-16T09:08:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:09 smithi111 ceph-mon[29803]: pgmap v623: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-16T09:08:10.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:09 smithi111 ceph-mon[40056]: pgmap v623: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-16T09:08:12.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:11 smithi114 ceph-mon[33315]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 853 B/s wr, 2 op/s 2024-09-16T09:08:12.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:08:11 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:08:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:11 smithi111 ceph-mon[29803]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 853 B/s wr, 2 op/s 2024-09-16T09:08:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:11 smithi111 ceph-mon[40056]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 853 B/s wr, 2 op/s 2024-09-16T09:08:13.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:12 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:12 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:13.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:12 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:13.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:08:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:08:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:08:14.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:13 smithi114 ceph-mon[33315]: pgmap v625: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 792 B/s wr, 3 op/s 2024-09-16T09:08:14.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:13 smithi111 ceph-mon[29803]: pgmap v625: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 792 B/s wr, 3 op/s 2024-09-16T09:08:14.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:13 smithi111 ceph-mon[40056]: pgmap v625: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 792 B/s wr, 3 op/s 2024-09-16T09:08:16.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:15 smithi114 ceph-mon[33315]: pgmap v626: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 703 B/s wr, 2 op/s 2024-09-16T09:08:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:15 smithi111 ceph-mon[29803]: pgmap v626: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 703 B/s wr, 2 op/s 2024-09-16T09:08:16.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:15 smithi111 ceph-mon[40056]: pgmap v626: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 703 B/s wr, 2 op/s 2024-09-16T09:08:17.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:17.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:17.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:18.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:17 smithi111 ceph-mon[29803]: pgmap v627: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-09-16T09:08:18.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:17 smithi111 ceph-mon[40056]: pgmap v627: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-09-16T09:08:18.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:17 smithi114 ceph-mon[33315]: pgmap v627: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-09-16T09:08:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:19 smithi111 ceph-mon[29803]: pgmap v628: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 87 B/s wr, 1 op/s 2024-09-16T09:08:20.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:19 smithi111 ceph-mon[40056]: pgmap v628: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 87 B/s wr, 1 op/s 2024-09-16T09:08:20.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:19 smithi114 ceph-mon[33315]: pgmap v628: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 87 B/s wr, 1 op/s 2024-09-16T09:08:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:21 smithi111 ceph-mon[29803]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-09-16T09:08:22.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:21 smithi111 ceph-mon[40056]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-09-16T09:08:22.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:21 smithi114 ceph-mon[33315]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-09-16T09:08:22.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:08:21 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:08:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:23.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:23.314 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:23.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:08:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:08:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:08:24.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:23 smithi114 ceph-mon[33315]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-09-16T09:08:24.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:23 smithi111 ceph-mon[40056]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-09-16T09:08:24.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:23 smithi111 ceph-mon[29803]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-09-16T09:08:26.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:25 smithi114 ceph-mon[33315]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:08:26.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:25 smithi111 ceph-mon[40056]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:08:26.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:25 smithi111 ceph-mon[29803]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:08:27.176 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0[43720]: 2024-09-16T09:08:26.920+0000 7fd979418640 -1 snap_mapper.add_oid found existing snaps mapped on 304:e38f5924:test-rados-api-smithi111-76716-80::foo:2, removing 2024-09-16T09:08:27.177 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2[55932]: 2024-09-16T09:08:26.920+0000 7f0b65b54640 -1 snap_mapper.add_oid found existing snaps mapped on 304:e38f5924:test-rados-api-smithi111-76716-80::foo:2, removing 2024-09-16T09:08:27.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:27.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:27.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-79"} : dispatch 2024-09-16T09:08:27.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-79"} : dispatch 2024-09-16T09:08:27.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:27.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:27.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-79"} : dispatch 2024-09-16T09:08:27.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:26 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-79"} : dispatch 2024-09-16T09:08:27.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:27.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:27.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-79"} : dispatch 2024-09-16T09:08:27.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:26 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-79"} : dispatch 2024-09-16T09:08:27.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:08:26 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:08:26.920+0000 7ff59dc31640 -1 snap_mapper.add_oid found existing snaps mapped on 304:e38f5924:test-rados-api-smithi111-76716-80::foo:2, removing 2024-09-16T09:08:28.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:27 smithi114 ceph-mon[33315]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:08:28.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:27 smithi111 ceph-mon[29803]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:08:28.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:27 smithi111 ceph-mon[40056]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:08:29.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:28 smithi114 ceph-mon[33315]: osdmap e425: 8 total, 8 up, 8 in 2024-09-16T09:08:29.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:28 smithi111 ceph-mon[29803]: osdmap e425: 8 total, 8 up, 8 in 2024-09-16T09:08:29.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:28 smithi111 ceph-mon[40056]: osdmap e425: 8 total, 8 up, 8 in 2024-09-16T09:08:30.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:29 smithi114 ceph-mon[33315]: pgmap v634: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:08:30.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:29 smithi114 ceph-mon[33315]: osdmap e426: 8 total, 8 up, 8 in 2024-09-16T09:08:30.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-81","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:30.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:29 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-81","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:30.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:29 smithi111 ceph-mon[29803]: pgmap v634: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:08:30.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:29 smithi111 ceph-mon[29803]: osdmap e426: 8 total, 8 up, 8 in 2024-09-16T09:08:30.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-81","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:30.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:29 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-81","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:30.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:29 smithi111 ceph-mon[40056]: pgmap v634: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:08:30.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:29 smithi111 ceph-mon[40056]: osdmap e426: 8 total, 8 up, 8 in 2024-09-16T09:08:30.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-81","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:30.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:29 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-81","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:31.327 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:31.327 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[29803]: osdmap e427: 8 total, 8 up, 8 in 2024-09-16T09:08:31.327 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:31.328 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:31.328 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:31.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:31.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[40056]: osdmap e427: 8 total, 8 up, 8 in 2024-09-16T09:08:31.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:31.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:31.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:31.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:31.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:30 smithi114 ceph-mon[33315]: osdmap e427: 8 total, 8 up, 8 in 2024-09-16T09:08:31.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:31.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:30 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:31.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:32.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:08:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:08:32.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:31 smithi114 ceph-mon[33315]: pgmap v637: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:08:32.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:08:32.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:31 smithi114 ceph-mon[33315]: osdmap e428: 8 total, 8 up, 8 in 2024-09-16T09:08:32.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:32.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:31 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:32.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:32.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[29803]: pgmap v637: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:08:32.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:08:32.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[29803]: osdmap e428: 8 total, 8 up, 8 in 2024-09-16T09:08:32.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:32.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:32.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:32.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[40056]: pgmap v637: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:08:32.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:08:32.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[40056]: osdmap e428: 8 total, 8 up, 8 in 2024-09-16T09:08:32.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:32.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:32.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:08:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:32 smithi114 ceph-mon[33315]: osdmap e429: 8 total, 8 up, 8 in 2024-09-16T09:08:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:33.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:08:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:08:33.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:33.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:08:33.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[29803]: osdmap e429: 8 total, 8 up, 8 in 2024-09-16T09:08:33.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:33.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:08:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[40056]: osdmap e429: 8 total, 8 up, 8 in 2024-09-16T09:08:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:34.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:33 smithi114 ceph-mon[33315]: pgmap v640: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:08:34.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:08:34.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:33 smithi114 ceph-mon[33315]: osdmap e430: 8 total, 8 up, 8 in 2024-09-16T09:08:34.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:34.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[29803]: pgmap v640: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:08:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:08:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[29803]: osdmap e430: 8 total, 8 up, 8 in 2024-09-16T09:08:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:34.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[40056]: pgmap v640: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:08:34.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:08:34.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[40056]: osdmap e430: 8 total, 8 up, 8 in 2024-09-16T09:08:34.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:34.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:35.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:35.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:34 smithi114 ceph-mon[33315]: osdmap e431: 8 total, 8 up, 8 in 2024-09-16T09:08:35.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "512"} : dispatch 2024-09-16T09:08:35.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:34 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "512"} : dispatch 2024-09-16T09:08:35.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:35.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:34 smithi111 ceph-mon[29803]: osdmap e431: 8 total, 8 up, 8 in 2024-09-16T09:08:35.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "512"} : dispatch 2024-09-16T09:08:35.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:34 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "512"} : dispatch 2024-09-16T09:08:35.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:35.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:34 smithi111 ceph-mon[40056]: osdmap e431: 8 total, 8 up, 8 in 2024-09-16T09:08:35.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "512"} : dispatch 2024-09-16T09:08:35.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:34 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "512"} : dispatch 2024-09-16T09:08:36.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:36 smithi114 ceph-mon[33315]: pgmap v643: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:08:36.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-09-16T09:08:36.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:36 smithi114 ceph-mon[33315]: osdmap e432: 8 total, 8 up, 8 in 2024-09-16T09:08:36.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "16384"} : dispatch 2024-09-16T09:08:36.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "16384"} : dispatch 2024-09-16T09:08:36.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[29803]: pgmap v643: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:08:36.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-09-16T09:08:36.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[29803]: osdmap e432: 8 total, 8 up, 8 in 2024-09-16T09:08:36.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "16384"} : dispatch 2024-09-16T09:08:36.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "16384"} : dispatch 2024-09-16T09:08:36.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[40056]: pgmap v643: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T09:08:36.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-09-16T09:08:36.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[40056]: osdmap e432: 8 total, 8 up, 8 in 2024-09-16T09:08:36.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "16384"} : dispatch 2024-09-16T09:08:36.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "16384"} : dispatch 2024-09-16T09:08:37.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-09-16T09:08:37.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:37 smithi114 ceph-mon[33315]: osdmap e433: 8 total, 8 up, 8 in 2024-09-16T09:08:37.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:37.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:37.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-09-16T09:08:37.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:37 smithi111 ceph-mon[29803]: osdmap e433: 8 total, 8 up, 8 in 2024-09-16T09:08:37.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:37.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:37.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-09-16T09:08:37.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:37 smithi111 ceph-mon[40056]: osdmap e433: 8 total, 8 up, 8 in 2024-09-16T09:08:37.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:37.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:38 smithi114 ceph-mon[33315]: pgmap v646: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-09-16T09:08:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:38 smithi114 ceph-mon[33315]: osdmap e434: 8 total, 8 up, 8 in 2024-09-16T09:08:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-81"} : dispatch 2024-09-16T09:08:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-81"} : dispatch 2024-09-16T09:08:38.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[29803]: pgmap v646: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-09-16T09:08:38.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:38.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[29803]: osdmap e434: 8 total, 8 up, 8 in 2024-09-16T09:08:38.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:38.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:38.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-81"} : dispatch 2024-09-16T09:08:38.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-81"} : dispatch 2024-09-16T09:08:38.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[40056]: pgmap v646: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-09-16T09:08:38.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:38.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[40056]: osdmap e434: 8 total, 8 up, 8 in 2024-09-16T09:08:38.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:38.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:38.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-81"} : dispatch 2024-09-16T09:08:38.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-81"} : dispatch 2024-09-16T09:08:39.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:39 smithi114 ceph-mon[33315]: osdmap e435: 8 total, 8 up, 8 in 2024-09-16T09:08:39.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:39 smithi111 ceph-mon[29803]: osdmap e435: 8 total, 8 up, 8 in 2024-09-16T09:08:39.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:39 smithi111 ceph-mon[40056]: osdmap e435: 8 total, 8 up, 8 in 2024-09-16T09:08:40.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:40 smithi114 ceph-mon[33315]: pgmap v649: 196 pgs: 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-09-16T09:08:40.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:40 smithi114 ceph-mon[33315]: osdmap e436: 8 total, 8 up, 8 in 2024-09-16T09:08:40.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-83","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:40.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-83","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:40 smithi111 ceph-mon[29803]: pgmap v649: 196 pgs: 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-09-16T09:08:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:40 smithi111 ceph-mon[29803]: osdmap e436: 8 total, 8 up, 8 in 2024-09-16T09:08:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-83","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-83","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:40.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:40 smithi111 ceph-mon[40056]: pgmap v649: 196 pgs: 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-09-16T09:08:40.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:40 smithi111 ceph-mon[40056]: osdmap e436: 8 total, 8 up, 8 in 2024-09-16T09:08:40.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-83","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:40.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-83","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-mon[33315]: osdmap e437: 8 total, 8 up, 8 in 2024-09-16T09:08:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:08:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-mon[33315]: osdmap e438: 8 total, 8 up, 8 in 2024-09-16T09:08:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[29803]: osdmap e437: 8 total, 8 up, 8 in 2024-09-16T09:08:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:41.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:08:41.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[29803]: osdmap e438: 8 total, 8 up, 8 in 2024-09-16T09:08:41.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:41.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:41.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:41.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[40056]: osdmap e437: 8 total, 8 up, 8 in 2024-09-16T09:08:41.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:41.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:41.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:41.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:08:41.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[40056]: osdmap e438: 8 total, 8 up, 8 in 2024-09-16T09:08:41.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:41.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:41 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:42.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:08:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:08:42.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:42 smithi114 ceph-mon[33315]: pgmap v652: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:08:42.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:08:42.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:42.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:42 smithi114 ceph-mon[33315]: osdmap e439: 8 total, 8 up, 8 in 2024-09-16T09:08:42.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[29803]: pgmap v652: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:08:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:08:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[29803]: osdmap e439: 8 total, 8 up, 8 in 2024-09-16T09:08:42.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:42.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[40056]: pgmap v652: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:08:42.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:08:42.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:42.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[40056]: osdmap e439: 8 total, 8 up, 8 in 2024-09-16T09:08:42.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:43.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:43 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:43.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:08:43.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:43 smithi114 ceph-mon[33315]: osdmap e440: 8 total, 8 up, 8 in 2024-09-16T09:08:43.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:43.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:43 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:43.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:08:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:08:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:08:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[29803]: osdmap e440: 8 total, 8 up, 8 in 2024-09-16T09:08:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:43.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:43.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:08:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[40056]: osdmap e440: 8 total, 8 up, 8 in 2024-09-16T09:08:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:43.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:43 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:44.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:44 smithi114 ceph-mon[33315]: pgmap v655: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-16T09:08:44.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:44.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:44 smithi114 ceph-mon[33315]: osdmap e441: 8 total, 8 up, 8 in 2024-09-16T09:08:44.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:44 smithi111 ceph-mon[29803]: pgmap v655: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-16T09:08:44.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:44.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:44 smithi111 ceph-mon[29803]: osdmap e441: 8 total, 8 up, 8 in 2024-09-16T09:08:44.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:44 smithi111 ceph-mon[40056]: pgmap v655: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-16T09:08:44.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:44.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:44 smithi111 ceph-mon[40056]: osdmap e441: 8 total, 8 up, 8 in 2024-09-16T09:08:46.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:46 smithi114 ceph-mon[33315]: pgmap v658: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-16T09:08:46.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:46 smithi114 ceph-mon[33315]: osdmap e442: 8 total, 8 up, 8 in 2024-09-16T09:08:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:46 smithi111 ceph-mon[29803]: pgmap v658: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-16T09:08:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:46 smithi111 ceph-mon[29803]: osdmap e442: 8 total, 8 up, 8 in 2024-09-16T09:08:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:46 smithi111 ceph-mon[40056]: pgmap v658: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-16T09:08:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:46 smithi111 ceph-mon[40056]: osdmap e442: 8 total, 8 up, 8 in 2024-09-16T09:08:47.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:47 smithi114 ceph-mon[33315]: osdmap e443: 8 total, 8 up, 8 in 2024-09-16T09:08:47.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:47.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:47 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:47.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-83"} : dispatch 2024-09-16T09:08:47.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:47 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-83"} : dispatch 2024-09-16T09:08:47.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:47.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[29803]: osdmap e443: 8 total, 8 up, 8 in 2024-09-16T09:08:47.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:47.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:47.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-83"} : dispatch 2024-09-16T09:08:47.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-83"} : dispatch 2024-09-16T09:08:47.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:47.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[40056]: osdmap e443: 8 total, 8 up, 8 in 2024-09-16T09:08:47.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:47.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:47.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-83"} : dispatch 2024-09-16T09:08:47.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-83"} : dispatch 2024-09-16T09:08:47.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:08:48.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:48 smithi114 ceph-mon[33315]: pgmap v661: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6.7 KiB/s wr, 12 op/s 2024-09-16T09:08:48.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:48 smithi114 ceph-mon[33315]: osdmap e444: 8 total, 8 up, 8 in 2024-09-16T09:08:48.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:48 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:08:48.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:48 smithi111 ceph-mon[29803]: pgmap v661: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6.7 KiB/s wr, 12 op/s 2024-09-16T09:08:48.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:48 smithi111 ceph-mon[29803]: osdmap e444: 8 total, 8 up, 8 in 2024-09-16T09:08:48.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:48 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:08:48.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:48 smithi111 ceph-mon[40056]: pgmap v661: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6.7 KiB/s wr, 12 op/s 2024-09-16T09:08:48.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:48 smithi111 ceph-mon[40056]: osdmap e444: 8 total, 8 up, 8 in 2024-09-16T09:08:48.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:48 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:08:49.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:49 smithi114 ceph-mon[33315]: osdmap e445: 8 total, 8 up, 8 in 2024-09-16T09:08:49.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-85","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:49.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:49 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-85","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:49.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:08:49.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:08:49.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:08:49.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[40056]: osdmap e445: 8 total, 8 up, 8 in 2024-09-16T09:08:49.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-85","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:49.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-85","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:49.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:08:49.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:08:49.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:08:49.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[29803]: osdmap e445: 8 total, 8 up, 8 in 2024-09-16T09:08:49.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-85","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:49.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-85","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:49.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:08:49.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:08:49.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:08:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:50 smithi114 ceph-mon[33315]: pgmap v664: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-09-16T09:08:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:50 smithi114 ceph-mon[33315]: osdmap e446: 8 total, 8 up, 8 in 2024-09-16T09:08:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:50 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:50 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:50.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[29803]: pgmap v664: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-09-16T09:08:50.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[29803]: osdmap e446: 8 total, 8 up, 8 in 2024-09-16T09:08:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:50.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[40056]: pgmap v664: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-09-16T09:08:50.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:50.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[40056]: osdmap e446: 8 total, 8 up, 8 in 2024-09-16T09:08:50.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:50.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:50.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:08:50.429 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:50 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:08:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:51 smithi114 ceph-mon[33315]: osdmap e447: 8 total, 8 up, 8 in 2024-09-16T09:08:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:08:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:51 smithi114 ceph-mon[33315]: osdmap e448: 8 total, 8 up, 8 in 2024-09-16T09:08:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:51 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:51.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:08:51.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:51.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[40056]: osdmap e447: 8 total, 8 up, 8 in 2024-09-16T09:08:51.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:51.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[40056]: osdmap e448: 8 total, 8 up, 8 in 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[29803]: osdmap e447: 8 total, 8 up, 8 in 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_tier","val": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[29803]: osdmap e448: 8 total, 8 up, 8 in 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:51 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:08:52.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:08:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:08:52.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:52 smithi114 ceph-mon[33315]: pgmap v667: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:08:52.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:08:52.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:52 smithi114 ceph-mon[33315]: osdmap e449: 8 total, 8 up, 8 in 2024-09-16T09:08:52.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:52.416 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:52.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[29803]: pgmap v667: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:08:52.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:08:52.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[29803]: osdmap e449: 8 total, 8 up, 8 in 2024-09-16T09:08:52.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:52.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:52.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[40056]: pgmap v667: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:08:52.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:08:52.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[40056]: osdmap e449: 8 total, 8 up, 8 in 2024-09-16T09:08:52.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:52.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:08:53.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:53.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:53.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:53 smithi114 ceph-mon[33315]: osdmap e450: 8 total, 8 up, 8 in 2024-09-16T09:08:53.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:53.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:53.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:53 smithi111 ceph-mon[40056]: osdmap e450: 8 total, 8 up, 8 in 2024-09-16T09:08:53.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:08:53.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:08:53.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:53 smithi111 ceph-mon[29803]: osdmap e450: 8 total, 8 up, 8 in 2024-09-16T09:08:53.427 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:08:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:08:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:08:54.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:54 smithi114 ceph-mon[33315]: pgmap v670: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:08:54.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:54 smithi111 ceph-mon[29803]: pgmap v670: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:08:54.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:54 smithi111 ceph-mon[40056]: pgmap v670: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:08:55.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:55 smithi114 ceph-mon[33315]: osdmap e451: 8 total, 8 up, 8 in 2024-09-16T09:08:55.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:55 smithi111 ceph-mon[40056]: osdmap e451: 8 total, 8 up, 8 in 2024-09-16T09:08:55.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:55 smithi111 ceph-mon[29803]: osdmap e451: 8 total, 8 up, 8 in 2024-09-16T09:08:56.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:56 smithi114 ceph-mon[33315]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:08:56.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:56 smithi114 ceph-mon[33315]: osdmap e452: 8 total, 8 up, 8 in 2024-09-16T09:08:56.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:56.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:56.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-85"} : dispatch 2024-09-16T09:08:56.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:56 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-85"} : dispatch 2024-09-16T09:08:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[29803]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:08:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[29803]: osdmap e452: 8 total, 8 up, 8 in 2024-09-16T09:08:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-85"} : dispatch 2024-09-16T09:08:56.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-85"} : dispatch 2024-09-16T09:08:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[40056]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:08:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[40056]: osdmap e452: 8 total, 8 up, 8 in 2024-09-16T09:08:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:08:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-85"} : dispatch 2024-09-16T09:08:56.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:56 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-85"} : dispatch 2024-09-16T09:08:57.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:57 smithi114 ceph-mon[33315]: osdmap e453: 8 total, 8 up, 8 in 2024-09-16T09:08:57.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:57 smithi111 ceph-mon[29803]: osdmap e453: 8 total, 8 up, 8 in 2024-09-16T09:08:57.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:57 smithi111 ceph-mon[40056]: osdmap e453: 8 total, 8 up, 8 in 2024-09-16T09:08:58.150 INFO:tasks.workunit.client.0.smithi111.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18098 ms) 2024-09-16T09:08:58.150 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-16T09:08:58.150 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22634 ms) 2024-09-16T09:08:58.150 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-16T09:08:58.150 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3019 ms) 2024-09-16T09:08:58.151 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-16T09:08:58.151 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.151 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3032 ms) 2024-09-16T09:08:58.151 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-16T09:08:58.151 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3020 ms) 2024-09-16T09:08:58.151 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-16T09:08:58.151 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.151 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3016 ms) 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4041 ms) 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39866 ms) 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16983 ms) 2024-09-16T09:08:58.152 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4030 ms) 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3017 ms) 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24357 ms) 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.153 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15061 ms) 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5053 ms) 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4040 ms) 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T09:08:58.154 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: done waiting 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25087 ms) 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10084 ms) 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9027 ms) 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9071 ms) 2024-09-16T09:08:58.155 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-16T09:08:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:58 smithi114 ceph-mon[33315]: pgmap v676: 196 pgs: 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-09-16T09:08:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:58 smithi114 ceph-mon[33315]: osdmap e454: 8 total, 8 up, 8 in 2024-09-16T09:08:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-87","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-87","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:58 smithi111 ceph-mon[40056]: pgmap v676: 196 pgs: 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-09-16T09:08:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:58 smithi111 ceph-mon[40056]: osdmap e454: 8 total, 8 up, 8 in 2024-09-16T09:08:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-87","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-87","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:58 smithi111 ceph-mon[29803]: pgmap v676: 196 pgs: 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-09-16T09:08:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:58 smithi111 ceph-mon[29803]: osdmap e454: 8 total, 8 up, 8 in 2024-09-16T09:08:58.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-87","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:58.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-87","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:08:59.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:59 smithi114 ceph-mon[33315]: osdmap e455: 8 total, 8 up, 8 in 2024-09-16T09:08:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:08:59 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:59 smithi111 ceph-mon[29803]: osdmap e455: 8 total, 8 up, 8 in 2024-09-16T09:08:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:59.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:08:59 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:08:59.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:08:59.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:59 smithi111 ceph-mon[40056]: osdmap e455: 8 total, 8 up, 8 in 2024-09-16T09:08:59.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:08:59.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:08:59 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:00 smithi114 ceph-mon[33315]: pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-09-16T09:09:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:00 smithi114 ceph-mon[33315]: osdmap e456: 8 total, 8 up, 8 in 2024-09-16T09:09:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:00 smithi111 ceph-mon[29803]: pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-09-16T09:09:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:00 smithi111 ceph-mon[29803]: osdmap e456: 8 total, 8 up, 8 in 2024-09-16T09:09:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:00.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:00.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:00 smithi111 ceph-mon[40056]: pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-09-16T09:09:00.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:00 smithi111 ceph-mon[40056]: osdmap e456: 8 total, 8 up, 8 in 2024-09-16T09:09:00.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:00.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:01.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:09:01.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:01 smithi114 ceph-mon[33315]: osdmap e457: 8 total, 8 up, 8 in 2024-09-16T09:09:01.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_tier","val": "test-rados-api-smithi111-76716-89-test-flush"} : dispatch 2024-09-16T09:09:01.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_tier","val": "test-rados-api-smithi111-76716-89-test-flush"} : dispatch 2024-09-16T09:09:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:09:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:01 smithi111 ceph-mon[29803]: osdmap e457: 8 total, 8 up, 8 in 2024-09-16T09:09:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_tier","val": "test-rados-api-smithi111-76716-89-test-flush"} : dispatch 2024-09-16T09:09:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_tier","val": "test-rados-api-smithi111-76716-89-test-flush"} : dispatch 2024-09-16T09:09:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:09:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:01 smithi111 ceph-mon[40056]: osdmap e457: 8 total, 8 up, 8 in 2024-09-16T09:09:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_tier","val": "test-rados-api-smithi111-76716-89-test-flush"} : dispatch 2024-09-16T09:09:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_tier","val": "test-rados-api-smithi111-76716-89-test-flush"} : dispatch 2024-09-16T09:09:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:02 smithi114 ceph-mon[33315]: pgmap v682: 260 pgs: 64 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_tier","val": "test-rados-api-smithi111-76716-89-test-flush"}]': finished 2024-09-16T09:09:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:09:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:02 smithi114 ceph-mon[33315]: osdmap e458: 8 total, 8 up, 8 in 2024-09-16T09:09:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:09:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:02 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:02.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:09:02 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:09:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[29803]: pgmap v682: 260 pgs: 64 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_tier","val": "test-rados-api-smithi111-76716-89-test-flush"}]': finished 2024-09-16T09:09:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:09:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[29803]: osdmap e458: 8 total, 8 up, 8 in 2024-09-16T09:09:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:09:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[40056]: pgmap v682: 260 pgs: 64 unknown, 196 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_tier","val": "test-rados-api-smithi111-76716-89-test-flush"}]': finished 2024-09-16T09:09:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:09:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[40056]: osdmap e458: 8 total, 8 up, 8 in 2024-09-16T09:09:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:09:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:02 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:03.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:09:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:09:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:09:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[29803]: osdmap e459: 8 total, 8 up, 8 in 2024-09-16T09:09:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:09:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:09:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:09:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[40056]: osdmap e459: 8 total, 8 up, 8 in 2024-09-16T09:09:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:09:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:09:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:03 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:09:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:03 smithi114 ceph-mon[33315]: osdmap e459: 8 total, 8 up, 8 in 2024-09-16T09:09:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:09:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:09:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:04 smithi114 ceph-mon[33315]: pgmap v685: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:09:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:09:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:04 smithi114 ceph-mon[33315]: osdmap e460: 8 total, 8 up, 8 in 2024-09-16T09:09:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:04 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:04 smithi111 ceph-mon[29803]: pgmap v685: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:09:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:09:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:04 smithi111 ceph-mon[29803]: osdmap e460: 8 total, 8 up, 8 in 2024-09-16T09:09:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:04 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:04.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:04 smithi111 ceph-mon[40056]: pgmap v685: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:09:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:09:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:04 smithi111 ceph-mon[40056]: osdmap e460: 8 total, 8 up, 8 in 2024-09-16T09:09:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:04 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:05 smithi114 ceph-mon[33315]: osdmap e461: 8 total, 8 up, 8 in 2024-09-16T09:09:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-87"} : dispatch 2024-09-16T09:09:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-87"} : dispatch 2024-09-16T09:09:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[29803]: osdmap e461: 8 total, 8 up, 8 in 2024-09-16T09:09:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-87"} : dispatch 2024-09-16T09:09:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-87"} : dispatch 2024-09-16T09:09:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[40056]: osdmap e461: 8 total, 8 up, 8 in 2024-09-16T09:09:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-87"} : dispatch 2024-09-16T09:09:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-87"} : dispatch 2024-09-16T09:09:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:06 smithi114 ceph-mon[33315]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:09:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:06 smithi114 ceph-mon[33315]: osdmap e462: 8 total, 8 up, 8 in 2024-09-16T09:09:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:06 smithi111 ceph-mon[29803]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:09:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:06 smithi111 ceph-mon[29803]: osdmap e462: 8 total, 8 up, 8 in 2024-09-16T09:09:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:06 smithi111 ceph-mon[40056]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:09:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:06 smithi111 ceph-mon[40056]: osdmap e462: 8 total, 8 up, 8 in 2024-09-16T09:09:07.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:07 smithi114 ceph-mon[33315]: osdmap e463: 8 total, 8 up, 8 in 2024-09-16T09:09:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-90","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:07 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-90","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:07 smithi111 ceph-mon[29803]: osdmap e463: 8 total, 8 up, 8 in 2024-09-16T09:09:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-90","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:07 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-90","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:07 smithi111 ceph-mon[40056]: osdmap e463: 8 total, 8 up, 8 in 2024-09-16T09:09:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-90","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:07.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:07 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-90","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:08 smithi114 ceph-mon[33315]: pgmap v691: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 22 op/s 2024-09-16T09:09:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:08 smithi114 ceph-mon[33315]: osdmap e464: 8 total, 8 up, 8 in 2024-09-16T09:09:08.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:08.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:08 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[29803]: pgmap v691: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 22 op/s 2024-09-16T09:09:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[29803]: osdmap e464: 8 total, 8 up, 8 in 2024-09-16T09:09:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:08.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[40056]: pgmap v691: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 22 op/s 2024-09-16T09:09:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[40056]: osdmap e464: 8 total, 8 up, 8 in 2024-09-16T09:09:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:08 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:09:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:09:09.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:09 smithi114 ceph-mon[33315]: osdmap e465: 8 total, 8 up, 8 in 2024-09-16T09:09:09.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:09:09.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:09 smithi111 ceph-mon[29803]: osdmap e465: 8 total, 8 up, 8 in 2024-09-16T09:09:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:09:09.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:09 smithi111 ceph-mon[40056]: osdmap e465: 8 total, 8 up, 8 in 2024-09-16T09:09:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:10 smithi114 ceph-mon[33315]: pgmap v694: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 22 op/s 2024-09-16T09:09:10.736 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:10 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:10.736 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:10 smithi114 ceph-mon[33315]: osdmap e466: 8 total, 8 up, 8 in 2024-09-16T09:09:10.737 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:10 smithi111 ceph-mon[29803]: pgmap v694: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 22 op/s 2024-09-16T09:09:10.737 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:10 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:10.737 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:10 smithi111 ceph-mon[29803]: osdmap e466: 8 total, 8 up, 8 in 2024-09-16T09:09:10.737 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:10 smithi111 ceph-mon[40056]: pgmap v694: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 22 op/s 2024-09-16T09:09:10.737 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:10 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:10.737 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:10 smithi111 ceph-mon[40056]: osdmap e466: 8 total, 8 up, 8 in 2024-09-16T09:09:11.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:11 smithi114 ceph-mon[33315]: osdmap e467: 8 total, 8 up, 8 in 2024-09-16T09:09:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-90"} : dispatch 2024-09-16T09:09:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-90"} : dispatch 2024-09-16T09:09:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[40056]: osdmap e467: 8 total, 8 up, 8 in 2024-09-16T09:09:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-90"} : dispatch 2024-09-16T09:09:11.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-90"} : dispatch 2024-09-16T09:09:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[29803]: osdmap e467: 8 total, 8 up, 8 in 2024-09-16T09:09:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-90"} : dispatch 2024-09-16T09:09:11.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-90"} : dispatch 2024-09-16T09:09:12.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:12 smithi114 ceph-mon[33315]: pgmap v697: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:12.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:12 smithi114 ceph-mon[33315]: osdmap e468: 8 total, 8 up, 8 in 2024-09-16T09:09:12.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:09:12 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:09:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:12 smithi111 ceph-mon[29803]: pgmap v697: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:12 smithi111 ceph-mon[29803]: osdmap e468: 8 total, 8 up, 8 in 2024-09-16T09:09:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:12 smithi111 ceph-mon[40056]: pgmap v697: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:12 smithi111 ceph-mon[40056]: osdmap e468: 8 total, 8 up, 8 in 2024-09-16T09:09:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:13 smithi114 ceph-mon[33315]: osdmap e469: 8 total, 8 up, 8 in 2024-09-16T09:09:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-92","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-92","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:13.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:09:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:09:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:09:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:13 smithi111 ceph-mon[29803]: osdmap e469: 8 total, 8 up, 8 in 2024-09-16T09:09:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-92","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-92","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:13 smithi111 ceph-mon[40056]: osdmap e469: 8 total, 8 up, 8 in 2024-09-16T09:09:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-92","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:13.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-92","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:14 smithi114 ceph-mon[33315]: pgmap v700: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 1.5 KiB/s wr, 161 op/s 2024-09-16T09:09:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:14 smithi114 ceph-mon[33315]: osdmap e470: 8 total, 8 up, 8 in 2024-09-16T09:09:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:14 smithi111 ceph-mon[40056]: pgmap v700: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 1.5 KiB/s wr, 161 op/s 2024-09-16T09:09:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:14 smithi111 ceph-mon[40056]: osdmap e470: 8 total, 8 up, 8 in 2024-09-16T09:09:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:14 smithi111 ceph-mon[29803]: pgmap v700: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 1.5 KiB/s wr, 161 op/s 2024-09-16T09:09:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:14 smithi111 ceph-mon[29803]: osdmap e470: 8 total, 8 up, 8 in 2024-09-16T09:09:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:15.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:15 smithi114 ceph-mon[33315]: osdmap e471: 8 total, 8 up, 8 in 2024-09-16T09:09:15.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:15 smithi111 ceph-mon[29803]: osdmap e471: 8 total, 8 up, 8 in 2024-09-16T09:09:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:15 smithi111 ceph-mon[40056]: osdmap e471: 8 total, 8 up, 8 in 2024-09-16T09:09:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:16 smithi114 ceph-mon[33315]: pgmap v703: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 1.5 KiB/s wr, 161 op/s 2024-09-16T09:09:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:16 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:16 smithi114 ceph-mon[33315]: osdmap e472: 8 total, 8 up, 8 in 2024-09-16T09:09:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-92"} : dispatch 2024-09-16T09:09:16.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:16 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-92"} : dispatch 2024-09-16T09:09:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[40056]: pgmap v703: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 1.5 KiB/s wr, 161 op/s 2024-09-16T09:09:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[40056]: osdmap e472: 8 total, 8 up, 8 in 2024-09-16T09:09:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-92"} : dispatch 2024-09-16T09:09:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-92"} : dispatch 2024-09-16T09:09:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[29803]: pgmap v703: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 1.5 KiB/s wr, 161 op/s 2024-09-16T09:09:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:09:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[29803]: osdmap e472: 8 total, 8 up, 8 in 2024-09-16T09:09:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-92"} : dispatch 2024-09-16T09:09:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:16 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-92"} : dispatch 2024-09-16T09:09:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:17 smithi114 ceph-mon[33315]: osdmap e473: 8 total, 8 up, 8 in 2024-09-16T09:09:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:17 smithi111 ceph-mon[29803]: osdmap e473: 8 total, 8 up, 8 in 2024-09-16T09:09:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:17 smithi111 ceph-mon[40056]: osdmap e473: 8 total, 8 up, 8 in 2024-09-16T09:09:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:18 smithi114 ceph-mon[33315]: pgmap v706: 196 pgs: 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:09:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:18 smithi114 ceph-mon[33315]: osdmap e474: 8 total, 8 up, 8 in 2024-09-16T09:09:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-94","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-94","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:18 smithi111 ceph-mon[29803]: pgmap v706: 196 pgs: 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:09:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:18 smithi111 ceph-mon[29803]: osdmap e474: 8 total, 8 up, 8 in 2024-09-16T09:09:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-94","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-94","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:18 smithi111 ceph-mon[40056]: pgmap v706: 196 pgs: 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:09:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:18 smithi111 ceph-mon[40056]: osdmap e474: 8 total, 8 up, 8 in 2024-09-16T09:09:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-94","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-94","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:19 smithi114 ceph-mon[33315]: osdmap e475: 8 total, 8 up, 8 in 2024-09-16T09:09:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:19 smithi111 ceph-mon[40056]: osdmap e475: 8 total, 8 up, 8 in 2024-09-16T09:09:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:19 smithi111 ceph-mon[29803]: osdmap e475: 8 total, 8 up, 8 in 2024-09-16T09:09:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:20 smithi114 ceph-mon[33315]: pgmap v709: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:09:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:20 smithi114 ceph-mon[33315]: osdmap e476: 8 total, 8 up, 8 in 2024-09-16T09:09:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:20 smithi111 ceph-mon[40056]: pgmap v709: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:09:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:20 smithi111 ceph-mon[40056]: osdmap e476: 8 total, 8 up, 8 in 2024-09-16T09:09:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:20 smithi111 ceph-mon[29803]: pgmap v709: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:09:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:20 smithi111 ceph-mon[29803]: osdmap e476: 8 total, 8 up, 8 in 2024-09-16T09:09:21.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:21 smithi114 ceph-mon[33315]: osdmap e477: 8 total, 8 up, 8 in 2024-09-16T09:09:21.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:21 smithi111 ceph-mon[40056]: osdmap e477: 8 total, 8 up, 8 in 2024-09-16T09:09:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:21 smithi111 ceph-mon[29803]: osdmap e477: 8 total, 8 up, 8 in 2024-09-16T09:09:22.308 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:09:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:09:22.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:22 smithi114 ceph-mon[33315]: pgmap v712: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:22 smithi111 ceph-mon[29803]: pgmap v712: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:22 smithi111 ceph-mon[40056]: pgmap v712: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:23.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:23 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:23.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:23 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:23 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:23.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:09:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:09:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:09:24.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:24 smithi114 ceph-mon[33315]: pgmap v713: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-16T09:09:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:24 smithi111 ceph-mon[29803]: pgmap v713: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-16T09:09:24.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:24 smithi111 ceph-mon[40056]: pgmap v713: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-16T09:09:26.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:26 smithi114 ceph-mon[33315]: pgmap v714: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-09-16T09:09:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:26 smithi111 ceph-mon[29803]: pgmap v714: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-09-16T09:09:26.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:26 smithi111 ceph-mon[40056]: pgmap v714: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-09-16T09:09:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:28 smithi114 ceph-mon[33315]: pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T09:09:28.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:28 smithi111 ceph-mon[29803]: pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T09:09:28.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:28 smithi111 ceph-mon[40056]: pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T09:09:30.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:30 smithi114 ceph-mon[33315]: pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 876 B/s wr, 3 op/s 2024-09-16T09:09:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:30 smithi111 ceph-mon[29803]: pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 876 B/s wr, 3 op/s 2024-09-16T09:09:30.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:30 smithi111 ceph-mon[40056]: pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 876 B/s wr, 3 op/s 2024-09-16T09:09:31.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:32.332 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:09:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:09:32.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:32 smithi114 ceph-mon[33315]: pgmap v717: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 792 B/s wr, 3 op/s 2024-09-16T09:09:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:32 smithi114 ceph-mon[33315]: osdmap e478: 8 total, 8 up, 8 in 2024-09-16T09:09:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:32 smithi111 ceph-mon[29803]: pgmap v717: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 792 B/s wr, 3 op/s 2024-09-16T09:09:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:32 smithi111 ceph-mon[29803]: osdmap e478: 8 total, 8 up, 8 in 2024-09-16T09:09:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:32 smithi111 ceph-mon[40056]: pgmap v717: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 792 B/s wr, 3 op/s 2024-09-16T09:09:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:32 smithi111 ceph-mon[40056]: osdmap e478: 8 total, 8 up, 8 in 2024-09-16T09:09:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:33 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:33.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:33 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:33 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:33.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:09:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:09:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:09:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:34 smithi114 ceph-mon[33315]: pgmap v719: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:34 smithi111 ceph-mon[29803]: pgmap v719: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:34.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:34 smithi111 ceph-mon[40056]: pgmap v719: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:36 smithi114 ceph-mon[33315]: pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:36 smithi111 ceph-mon[29803]: pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:36.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:36 smithi111 ceph-mon[40056]: pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:38.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:38 smithi114 ceph-mon[33315]: pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:38 smithi114 ceph-mon[33315]: osdmap e479: 8 total, 8 up, 8 in 2024-09-16T09:09:38.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:38 smithi111 ceph-mon[40056]: pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:38.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:38 smithi111 ceph-mon[40056]: osdmap e479: 8 total, 8 up, 8 in 2024-09-16T09:09:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:38 smithi111 ceph-mon[29803]: pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:38.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:38 smithi111 ceph-mon[29803]: osdmap e479: 8 total, 8 up, 8 in 2024-09-16T09:09:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:40 smithi114 ceph-mon[33315]: pgmap v723: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:40.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:40 smithi111 ceph-mon[40056]: pgmap v723: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:40 smithi111 ceph-mon[29803]: pgmap v723: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:42.356 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:09:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:09:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:42 smithi114 ceph-mon[33315]: pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-94"} : dispatch 2024-09-16T09:09:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:42 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-94"} : dispatch 2024-09-16T09:09:42.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[40056]: pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:42.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:42.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:42.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-94"} : dispatch 2024-09-16T09:09:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-94"} : dispatch 2024-09-16T09:09:42.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[29803]: pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:09:42.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:42.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:42.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-94"} : dispatch 2024-09-16T09:09:42.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:42 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-94"} : dispatch 2024-09-16T09:09:43.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:43 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:43 smithi114 ceph-mon[33315]: osdmap e480: 8 total, 8 up, 8 in 2024-09-16T09:09:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:43 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:43 smithi111 ceph-mon[29803]: osdmap e480: 8 total, 8 up, 8 in 2024-09-16T09:09:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:09:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:09:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:09:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:43 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:43 smithi111 ceph-mon[40056]: osdmap e480: 8 total, 8 up, 8 in 2024-09-16T09:09:44.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:44 smithi114 ceph-mon[33315]: pgmap v726: 196 pgs: 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:44 smithi114 ceph-mon[33315]: osdmap e481: 8 total, 8 up, 8 in 2024-09-16T09:09:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-96","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:44 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-96","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:44 smithi111 ceph-mon[29803]: pgmap v726: 196 pgs: 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:44 smithi111 ceph-mon[29803]: osdmap e481: 8 total, 8 up, 8 in 2024-09-16T09:09:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-96","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:44 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-96","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:44 smithi111 ceph-mon[40056]: pgmap v726: 196 pgs: 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:44 smithi111 ceph-mon[40056]: osdmap e481: 8 total, 8 up, 8 in 2024-09-16T09:09:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-96","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:44 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-96","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:45 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:45 smithi114 ceph-mon[33315]: osdmap e482: 8 total, 8 up, 8 in 2024-09-16T09:09:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:45.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:45 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:45.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:45 smithi111 ceph-mon[40056]: osdmap e482: 8 total, 8 up, 8 in 2024-09-16T09:09:45.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:45.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:45 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:45 smithi111 ceph-mon[29803]: osdmap e482: 8 total, 8 up, 8 in 2024-09-16T09:09:45.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:09:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:46 smithi114 ceph-mon[33315]: pgmap v729: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:46 smithi114 ceph-mon[33315]: osdmap e483: 8 total, 8 up, 8 in 2024-09-16T09:09:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:46 smithi111 ceph-mon[29803]: pgmap v729: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:46 smithi111 ceph-mon[29803]: osdmap e483: 8 total, 8 up, 8 in 2024-09-16T09:09:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:46 smithi111 ceph-mon[40056]: pgmap v729: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:09:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:46 smithi111 ceph-mon[40056]: osdmap e483: 8 total, 8 up, 8 in 2024-09-16T09:09:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:09:47.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:47 smithi114 ceph-mon[33315]: pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-16T09:09:47.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:47 smithi111 ceph-mon[29803]: pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-16T09:09:47.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:47 smithi111 ceph-mon[40056]: pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-16T09:09:48.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:48 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:09:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:48 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:09:48.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:48 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:09:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: pgmap v732: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-09-16T09:09:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi114 to 3248M 2024-09-16T09:09:49.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: Adjusting osd_memory_target on smithi111 to 3248M 2024-09-16T09:09:49.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:09:49.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:09:49.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:09:49.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:09:49.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:49 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:09:49.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: pgmap v732: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-09-16T09:09:49.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi114 to 3248M 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: Adjusting osd_memory_target on smithi111 to 3248M 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:09:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: pgmap v732: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi114 to 3248M 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: Adjusting osd_memory_target on smithi111 to 3248M 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:09:49.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:09:49.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:09:49.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:49 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:09:52.056 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:51 smithi114 ceph-mon[33315]: pgmap v733: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 703 B/s rd, 703 B/s wr, 2 op/s 2024-09-16T09:09:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:51 smithi111 ceph-mon[29803]: pgmap v733: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 703 B/s rd, 703 B/s wr, 2 op/s 2024-09-16T09:09:52.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:51 smithi111 ceph-mon[40056]: pgmap v733: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 703 B/s rd, 703 B/s wr, 2 op/s 2024-09-16T09:09:52.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:09:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:09:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:52 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:52 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:53.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:52 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:09:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:09:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:09:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:09:54.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:53 smithi114 ceph-mon[33315]: pgmap v734: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 618 B/s wr, 2 op/s 2024-09-16T09:09:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:53 smithi111 ceph-mon[29803]: pgmap v734: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 618 B/s wr, 2 op/s 2024-09-16T09:09:54.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:53 smithi111 ceph-mon[40056]: pgmap v734: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 618 B/s wr, 2 op/s 2024-09-16T09:09:55.676 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3[62019]: 2024-09-16T09:09:55.418+0000 7f0966ce6640 -1 snap_mapper.add_oid found existing snaps mapped on 107:bc3b9c83:test-rados-api-smithi111-76716-97::foo:21, removing 2024-09-16T09:09:55.708 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:09:55 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:09:55.418+0000 7f8994694640 -1 snap_mapper.add_oid found existing snaps mapped on 107:bc3b9c83:test-rados-api-smithi111-76716-97::foo:21, removing 2024-09-16T09:09:55.709 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:09:55 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:09:55.419+0000 7fbe6ff5c640 -1 snap_mapper.add_oid found existing snaps mapped on 107:bc3b9c83:test-rados-api-smithi111-76716-97::foo:21, removing 2024-09-16T09:09:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:55 smithi114 ceph-mon[33315]: pgmap v735: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:09:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-96"} : dispatch 2024-09-16T09:09:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:55 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-96"} : dispatch 2024-09-16T09:09:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[29803]: pgmap v735: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:09:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-96"} : dispatch 2024-09-16T09:09:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-96"} : dispatch 2024-09-16T09:09:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[40056]: pgmap v735: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:09:56.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:56.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:09:56.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-96"} : dispatch 2024-09-16T09:09:56.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:55 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-96"} : dispatch 2024-09-16T09:09:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:56 smithi114 ceph-mon[33315]: osdmap e484: 8 total, 8 up, 8 in 2024-09-16T09:09:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:56 smithi111 ceph-mon[29803]: osdmap e484: 8 total, 8 up, 8 in 2024-09-16T09:09:57.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:56 smithi111 ceph-mon[40056]: osdmap e484: 8 total, 8 up, 8 in 2024-09-16T09:09:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:57 smithi114 ceph-mon[33315]: pgmap v737: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-09-16T09:09:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:57 smithi114 ceph-mon[33315]: osdmap e485: 8 total, 8 up, 8 in 2024-09-16T09:09:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-98","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:57 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-98","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:57 smithi111 ceph-mon[29803]: pgmap v737: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-09-16T09:09:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:57 smithi111 ceph-mon[29803]: osdmap e485: 8 total, 8 up, 8 in 2024-09-16T09:09:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-98","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:57 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-98","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:57 smithi111 ceph-mon[40056]: pgmap v737: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-09-16T09:09:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:57 smithi111 ceph-mon[40056]: osdmap e485: 8 total, 8 up, 8 in 2024-09-16T09:09:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-98","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:57 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-98","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:09:59.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:58 smithi114 ceph-mon[33315]: osdmap e486: 8 total, 8 up, 8 in 2024-09-16T09:09:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:09:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:58 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:09:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:58 smithi111 ceph-mon[29803]: osdmap e486: 8 total, 8 up, 8 in 2024-09-16T09:09:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:09:59.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:58 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:09:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:09:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:58 smithi111 ceph-mon[40056]: osdmap e486: 8 total, 8 up, 8 in 2024-09-16T09:09:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:09:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:58 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:10:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:59 smithi114 ceph-mon[33315]: pgmap v740: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:10:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:10:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-98", "mode": "writeback"} : dispatch 2024-09-16T09:10:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:59 smithi114 ceph-mon[33315]: osdmap e487: 8 total, 8 up, 8 in 2024-09-16T09:10:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:09:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-98", "mode": "writeback"} : dispatch 2024-09-16T09:10:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[29803]: pgmap v740: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:10:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:10:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-98", "mode": "writeback"} : dispatch 2024-09-16T09:10:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[29803]: osdmap e487: 8 total, 8 up, 8 in 2024-09-16T09:10:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-98", "mode": "writeback"} : dispatch 2024-09-16T09:10:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[40056]: pgmap v740: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:10:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:10:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-98", "mode": "writeback"} : dispatch 2024-09-16T09:10:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[40056]: osdmap e487: 8 total, 8 up, 8 in 2024-09-16T09:10:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:09:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-98", "mode": "writeback"} : dispatch 2024-09-16T09:10:01.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:01.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-98", "mode": "writeback"}]': finished 2024-09-16T09:10:01.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: osdmap e488: 8 total, 8 up, 8 in 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 1 pool(s) do not have an application enabled 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: pool 'test-rados-api-smithi111-76716-98' with cache_mode writeback needs hit_set_type to be set but it is not 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-98"}]': finished 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: osdmap e489: 8 total, 8 up, 8 in 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:10:01.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-98", "mode": "writeback"}]': finished 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: osdmap e488: 8 total, 8 up, 8 in 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 1 pool(s) do not have an application enabled 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: pool 'test-rados-api-smithi111-76716-98' with cache_mode writeback needs hit_set_type to be set but it is not 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-98"}]': finished 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: osdmap e489: 8 total, 8 up, 8 in 2024-09-16T09:10:01.079 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:10:01.080 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:10:01.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-98", "mode": "writeback"}]': finished 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: osdmap e488: 8 total, 8 up, 8 in 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 1 pool(s) do not have an application enabled 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: pool 'test-rados-api-smithi111-76716-98' with cache_mode writeback needs hit_set_type to be set but it is not 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-16T09:10:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-98"}]': finished 2024-09-16T09:10:01.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: osdmap e489: 8 total, 8 up, 8 in 2024-09-16T09:10:01.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:10:01.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:10:02.349 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:10:02 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:10:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:02 smithi114 ceph-mon[33315]: pgmap v743: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:02 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:10:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:02 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:02 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:10:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:02 smithi114 ceph-mon[33315]: osdmap e490: 8 total, 8 up, 8 in 2024-09-16T09:10:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_count","val": "1"} : dispatch 2024-09-16T09:10:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_count","val": "1"} : dispatch 2024-09-16T09:10:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[29803]: pgmap v743: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:10:02.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:10:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[29803]: osdmap e490: 8 total, 8 up, 8 in 2024-09-16T09:10:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_count","val": "1"} : dispatch 2024-09-16T09:10:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_count","val": "1"} : dispatch 2024-09-16T09:10:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[40056]: pgmap v743: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:10:02.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:02.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:02.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:10:02.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[40056]: osdmap e490: 8 total, 8 up, 8 in 2024-09-16T09:10:02.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_count","val": "1"} : dispatch 2024-09-16T09:10:02.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_count","val": "1"} : dispatch 2024-09-16T09:10:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:03 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_count","val": "1"}]': finished 2024-09-16T09:10:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:03 smithi114 ceph-mon[33315]: osdmap e491: 8 total, 8 up, 8 in 2024-09-16T09:10:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:10:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:10:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_count","val": "1"}]': finished 2024-09-16T09:10:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[29803]: osdmap e491: 8 total, 8 up, 8 in 2024-09-16T09:10:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_count","val": "1"}]': finished 2024-09-16T09:10:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[40056]: osdmap e491: 8 total, 8 up, 8 in 2024-09-16T09:10:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:03.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:04 smithi114 ceph-mon[33315]: pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:10:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:10:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:04 smithi114 ceph-mon[33315]: osdmap e492: 8 total, 8 up, 8 in 2024-09-16T09:10:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "target_max_objects","val": "250"} : dispatch 2024-09-16T09:10:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "target_max_objects","val": "250"} : dispatch 2024-09-16T09:10:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[29803]: pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:10:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:10:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[29803]: osdmap e492: 8 total, 8 up, 8 in 2024-09-16T09:10:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "target_max_objects","val": "250"} : dispatch 2024-09-16T09:10:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "target_max_objects","val": "250"} : dispatch 2024-09-16T09:10:04.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[40056]: pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:10:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:10:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[40056]: osdmap e492: 8 total, 8 up, 8 in 2024-09-16T09:10:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "target_max_objects","val": "250"} : dispatch 2024-09-16T09:10:04.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "target_max_objects","val": "250"} : dispatch 2024-09-16T09:10:05.664 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:10:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=cleanup t=2024-09-16T09:10:05.27549398Z level=info msg="Completed cleanup jobs" duration=11.93613ms 2024-09-16T09:10:05.664 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:10:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore.transactions t=2024-09-16T09:10:05.281918881Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-16T09:10:05.664 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:10:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=plugins.update.checker t=2024-09-16T09:10:05.392558647Z level=info msg="Update check succeeded" duration=59.395945ms 2024-09-16T09:10:05.664 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:10:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=grafana.update.checker t=2024-09-16T09:10:05.411465894Z level=info msg="Update check succeeded" duration=69.582495ms 2024-09-16T09:10:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:05 smithi114 ceph-mon[33315]: pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:10:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "target_max_objects","val": "250"}]': finished 2024-09-16T09:10:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:05 smithi114 ceph-mon[33315]: osdmap e493: 8 total, 8 up, 8 in 2024-09-16T09:10:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:05 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[29803]: pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:10:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "target_max_objects","val": "250"}]': finished 2024-09-16T09:10:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[29803]: osdmap e493: 8 total, 8 up, 8 in 2024-09-16T09:10:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[40056]: pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:10:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-98","var": "target_max_objects","val": "250"}]': finished 2024-09-16T09:10:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[40056]: osdmap e493: 8 total, 8 up, 8 in 2024-09-16T09:10:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:05 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:10:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:06 smithi114 ceph-mon[33315]: osdmap e494: 8 total, 8 up, 8 in 2024-09-16T09:10:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:06 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98"}]': finished 2024-09-16T09:10:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:06 smithi114 ceph-mon[33315]: osdmap e495: 8 total, 8 up, 8 in 2024-09-16T09:10:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:10:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[29803]: osdmap e494: 8 total, 8 up, 8 in 2024-09-16T09:10:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98"}]': finished 2024-09-16T09:10:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[29803]: osdmap e495: 8 total, 8 up, 8 in 2024-09-16T09:10:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:10:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[40056]: osdmap e494: 8 total, 8 up, 8 in 2024-09-16T09:10:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98"} : dispatch 2024-09-16T09:10:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-98"}]': finished 2024-09-16T09:10:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:06 smithi111 ceph-mon[40056]: osdmap e495: 8 total, 8 up, 8 in 2024-09-16T09:10:08.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:07 smithi114 ceph-mon[33315]: pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:07 smithi111 ceph-mon[29803]: pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:08.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:07 smithi111 ceph-mon[40056]: pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:09.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:08 smithi114 ceph-mon[33315]: osdmap e496: 8 total, 8 up, 8 in 2024-09-16T09:10:09.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:08 smithi111 ceph-mon[29803]: osdmap e496: 8 total, 8 up, 8 in 2024-09-16T09:10:09.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:08 smithi111 ceph-mon[40056]: osdmap e496: 8 total, 8 up, 8 in 2024-09-16T09:10:10.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:09 smithi114 ceph-mon[33315]: pgmap v755: 196 pgs: 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:10.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:09 smithi114 ceph-mon[33315]: osdmap e497: 8 total, 8 up, 8 in 2024-09-16T09:10:10.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-100","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:10.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:09 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-100","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:09 smithi111 ceph-mon[29803]: pgmap v755: 196 pgs: 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:09 smithi111 ceph-mon[29803]: osdmap e497: 8 total, 8 up, 8 in 2024-09-16T09:10:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-100","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:09 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-100","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:10.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:09 smithi111 ceph-mon[40056]: pgmap v755: 196 pgs: 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:10.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:09 smithi111 ceph-mon[40056]: osdmap e497: 8 total, 8 up, 8 in 2024-09-16T09:10:10.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-100","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:10.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:09 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-100","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:11.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:11.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:10 smithi114 ceph-mon[33315]: osdmap e498: 8 total, 8 up, 8 in 2024-09-16T09:10:11.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:10:11.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:10 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:10:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:10 smithi111 ceph-mon[29803]: osdmap e498: 8 total, 8 up, 8 in 2024-09-16T09:10:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:10:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:10 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:10:11.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:11.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:10 smithi111 ceph-mon[40056]: osdmap e498: 8 total, 8 up, 8 in 2024-09-16T09:10:11.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:10:11.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:10 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:10:12.078 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:11 smithi114 ceph-mon[33315]: pgmap v758: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:12.078 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:10:12.078 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:11 smithi114 ceph-mon[33315]: osdmap e499: 8 total, 8 up, 8 in 2024-09-16T09:10:12.078 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:12.078 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:11 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[29803]: pgmap v758: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:10:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[29803]: osdmap e499: 8 total, 8 up, 8 in 2024-09-16T09:10:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:12.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[40056]: pgmap v758: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:10:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[40056]: osdmap e499: 8 total, 8 up, 8 in 2024-09-16T09:10:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:11 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:12.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:10:12 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:10:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-100"}]': finished 2024-09-16T09:10:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:12 smithi114 ceph-mon[33315]: osdmap e500: 8 total, 8 up, 8 in 2024-09-16T09:10:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-100", "mode": "writeback"} : dispatch 2024-09-16T09:10:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-100", "mode": "writeback"} : dispatch 2024-09-16T09:10:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:12 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-100"}]': finished 2024-09-16T09:10:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[29803]: osdmap e500: 8 total, 8 up, 8 in 2024-09-16T09:10:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-100", "mode": "writeback"} : dispatch 2024-09-16T09:10:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-100", "mode": "writeback"} : dispatch 2024-09-16T09:10:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:13.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-6", "overlaypool": "test-rados-api-smithi111-76716-100"}]': finished 2024-09-16T09:10:13.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[40056]: osdmap e500: 8 total, 8 up, 8 in 2024-09-16T09:10:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-100", "mode": "writeback"} : dispatch 2024-09-16T09:10:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-100", "mode": "writeback"} : dispatch 2024-09-16T09:10:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:12 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:13.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:10:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:10:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:13 smithi114 ceph-mon[33315]: pgmap v761: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:13 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-100", "mode": "writeback"}]': finished 2024-09-16T09:10:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:10:14.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:13 smithi114 ceph-mon[33315]: osdmap e501: 8 total, 8 up, 8 in 2024-09-16T09:10:14.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:10:14.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[29803]: pgmap v761: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:14.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:14.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-100", "mode": "writeback"}]': finished 2024-09-16T09:10:14.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:10:14.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[29803]: osdmap e501: 8 total, 8 up, 8 in 2024-09-16T09:10:14.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:10:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[40056]: pgmap v761: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-100", "mode": "writeback"}]': finished 2024-09-16T09:10:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:10:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[40056]: osdmap e501: 8 total, 8 up, 8 in 2024-09-16T09:10:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:10:15.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:10:15.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:14 smithi114 ceph-mon[33315]: osdmap e502: 8 total, 8 up, 8 in 2024-09-16T09:10:15.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:15.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:15.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:10:15.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:14 smithi114 ceph-mon[33315]: osdmap e503: 8 total, 8 up, 8 in 2024-09-16T09:10:15.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:10:15.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:14 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:10:15.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:10:15.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[29803]: osdmap e502: 8 total, 8 up, 8 in 2024-09-16T09:10:15.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:15.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:15.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[29803]: osdmap e503: 8 total, 8 up, 8 in 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[40056]: osdmap e502: 8 total, 8 up, 8 in 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[40056]: osdmap e503: 8 total, 8 up, 8 in 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:10:15.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:14 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:10:16.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:15 smithi114 ceph-mon[33315]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:16.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:15 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:16.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:10:16.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:15 smithi114 ceph-mon[33315]: osdmap e504: 8 total, 8 up, 8 in 2024-09-16T09:10:16.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "min_read_recency_for_promote","val": "10000"} : dispatch 2024-09-16T09:10:16.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "min_read_recency_for_promote","val": "10000"} : dispatch 2024-09-16T09:10:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[29803]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:10:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[29803]: osdmap e504: 8 total, 8 up, 8 in 2024-09-16T09:10:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "min_read_recency_for_promote","val": "10000"} : dispatch 2024-09-16T09:10:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "min_read_recency_for_promote","val": "10000"} : dispatch 2024-09-16T09:10:16.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[40056]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:16.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:16.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:10:16.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[40056]: osdmap e504: 8 total, 8 up, 8 in 2024-09-16T09:10:16.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "min_read_recency_for_promote","val": "10000"} : dispatch 2024-09-16T09:10:16.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "min_read_recency_for_promote","val": "10000"} : dispatch 2024-09-16T09:10:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:10:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-09-16T09:10:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:17 smithi114 ceph-mon[33315]: osdmap e505: 8 total, 8 up, 8 in 2024-09-16T09:10:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:17 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:10:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-09-16T09:10:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[29803]: osdmap e505: 8 total, 8 up, 8 in 2024-09-16T09:10:17.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:17.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:10:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-09-16T09:10:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[40056]: osdmap e505: 8 total, 8 up, 8 in 2024-09-16T09:10:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:17.679 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:17 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:18 smithi114 ceph-mon[33315]: pgmap v767: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:10:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:18 smithi114 ceph-mon[33315]: osdmap e506: 8 total, 8 up, 8 in 2024-09-16T09:10:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:18 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[29803]: pgmap v767: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:10:18.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[29803]: osdmap e506: 8 total, 8 up, 8 in 2024-09-16T09:10:18.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:18.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[40056]: pgmap v767: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"}]': finished 2024-09-16T09:10:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[40056]: osdmap e506: 8 total, 8 up, 8 in 2024-09-16T09:10:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:18 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100"} : dispatch 2024-09-16T09:10:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:19 smithi114 ceph-mon[33315]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100"}]': finished 2024-09-16T09:10:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:19 smithi114 ceph-mon[33315]: osdmap e507: 8 total, 8 up, 8 in 2024-09-16T09:10:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:19 smithi111 ceph-mon[29803]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100"}]': finished 2024-09-16T09:10:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:19 smithi111 ceph-mon[29803]: osdmap e507: 8 total, 8 up, 8 in 2024-09-16T09:10:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:19 smithi111 ceph-mon[40056]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-100"}]': finished 2024-09-16T09:10:20.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:19 smithi111 ceph-mon[40056]: osdmap e507: 8 total, 8 up, 8 in 2024-09-16T09:10:21.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:20 smithi114 ceph-mon[33315]: osdmap e508: 8 total, 8 up, 8 in 2024-09-16T09:10:21.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:20 smithi114 ceph-mon[33315]: osdmap e509: 8 total, 8 up, 8 in 2024-09-16T09:10:21.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-102","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:21.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-102","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:21.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:20 smithi111 ceph-mon[40056]: osdmap e508: 8 total, 8 up, 8 in 2024-09-16T09:10:21.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:20 smithi111 ceph-mon[40056]: osdmap e509: 8 total, 8 up, 8 in 2024-09-16T09:10:21.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-102","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:21.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-102","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:21.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:20 smithi111 ceph-mon[29803]: osdmap e508: 8 total, 8 up, 8 in 2024-09-16T09:10:21.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:20 smithi111 ceph-mon[29803]: osdmap e509: 8 total, 8 up, 8 in 2024-09-16T09:10:21.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-102","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:21.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-102","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:22.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:10:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:10:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:21 smithi114 ceph-mon[33315]: pgmap v773: 196 pgs: 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:21 smithi114 ceph-mon[33315]: osdmap e510: 8 total, 8 up, 8 in 2024-09-16T09:10:22.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:10:22.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:10:22.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:10:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[29803]: pgmap v773: 196 pgs: 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[29803]: osdmap e510: 8 total, 8 up, 8 in 2024-09-16T09:10:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:10:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:10:22.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:10:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[40056]: pgmap v773: 196 pgs: 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[40056]: osdmap e510: 8 total, 8 up, 8 in 2024-09-16T09:10:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd dump","format":"json"} : dispatch 2024-09-16T09:10:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:10:22.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "fingerprint_algorithm","val": "sha1"} : dispatch 2024-09-16T09:10:23.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:23.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:10:23.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:22 smithi114 ceph-mon[33315]: osdmap e511: 8 total, 8 up, 8 in 2024-09-16T09:10:23.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:10:23.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:10:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:10:23.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[29803]: osdmap e511: 8 total, 8 up, 8 in 2024-09-16T09:10:23.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:10:23.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:10:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-16T09:10:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[40056]: osdmap e511: 8 total, 8 up, 8 in 2024-09-16T09:10:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:10:23.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_chunk_algorithm","val": "fastcdc"} : dispatch 2024-09-16T09:10:23.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:10:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:10:24.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:23 smithi114 ceph-mon[33315]: pgmap v776: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:24.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:10:24.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:23 smithi114 ceph-mon[33315]: osdmap e512: 8 total, 8 up, 8 in 2024-09-16T09:10:24.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:10:24.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:10:24.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[40056]: pgmap v776: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:24.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:10:24.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[40056]: osdmap e512: 8 total, 8 up, 8 in 2024-09-16T09:10:24.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:10:24.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:10:24.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[29803]: pgmap v776: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:24.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-16T09:10:24.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[29803]: osdmap e512: 8 total, 8 up, 8 in 2024-09-16T09:10:24.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:10:24.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_cdc_chunk_size","val": "1024"} : dispatch 2024-09-16T09:10:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:25 smithi114 ceph-mon[33315]: pgmap v779: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:10:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:25 smithi114 ceph-mon[33315]: osdmap e513: 8 total, 8 up, 8 in 2024-09-16T09:10:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-102"} : dispatch 2024-09-16T09:10:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:25 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-102"} : dispatch 2024-09-16T09:10:26.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[29803]: pgmap v779: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:26.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:10:26.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[29803]: osdmap e513: 8 total, 8 up, 8 in 2024-09-16T09:10:26.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-102"} : dispatch 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-102"} : dispatch 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[40056]: pgmap v779: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[40056]: osdmap e513: 8 total, 8 up, 8 in 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-6"} : dispatch 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2791171118' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-102"} : dispatch 2024-09-16T09:10:26.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:25 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-6", "tierpool": "test-rados-api-smithi111-76716-102"} : dispatch 2024-09-16T09:10:27.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:26 smithi114 ceph-mon[33315]: osdmap e514: 8 total, 8 up, 8 in 2024-09-16T09:10:27.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:26 smithi111 ceph-mon[29803]: osdmap e514: 8 total, 8 up, 8 in 2024-09-16T09:10:27.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:26 smithi111 ceph-mon[40056]: osdmap e514: 8 total, 8 up, 8 in 2024-09-16T09:10:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:27 smithi114 ceph-mon[33315]: pgmap v782: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:10:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:27 smithi114 ceph-mon[33315]: osdmap e515: 8 total, 8 up, 8 in 2024-09-16T09:10:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:10:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:27 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:10:28.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[29803]: pgmap v782: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:10:28.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[29803]: osdmap e515: 8 total, 8 up, 8 in 2024-09-16T09:10:28.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[40056]: pgmap v782: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[40056]: osdmap e515: 8 total, 8 up, 8 in 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:28.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:10:28.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:27 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:10:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:10:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:28 smithi114 ceph-mon[33315]: osdmap e516: 8 total, 8 up, 8 in 2024-09-16T09:10:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi111-76716-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:29.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:28 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi111-76716-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:29.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:10:29.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:28 smithi111 ceph-mon[40056]: osdmap e516: 8 total, 8 up, 8 in 2024-09-16T09:10:29.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi111-76716-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:29.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:28 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi111-76716-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:29.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:10:29.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:28 smithi111 ceph-mon[29803]: osdmap e516: 8 total, 8 up, 8 in 2024-09-16T09:10:29.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi111-76716-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:29.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:28 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi111-76716-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:10:30.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:29 smithi111 ceph-mon[29803]: pgmap v785: 164 pgs: 164 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:30.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:29 smithi111 ceph-mon[29803]: osdmap e517: 8 total, 8 up, 8 in 2024-09-16T09:10:30.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:29 smithi111 ceph-mon[40056]: pgmap v785: 164 pgs: 164 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:30.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:29 smithi111 ceph-mon[40056]: osdmap e517: 8 total, 8 up, 8 in 2024-09-16T09:10:30.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:29 smithi114 ceph-mon[33315]: pgmap v785: 164 pgs: 164 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:30.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:29 smithi114 ceph-mon[33315]: osdmap e517: 8 total, 8 up, 8 in 2024-09-16T09:10:31.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:30 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi111-76716-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi111-76716-104"}]': finished 2024-09-16T09:10:31.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:30 smithi114 ceph-mon[33315]: osdmap e518: 8 total, 8 up, 8 in 2024-09-16T09:10:31.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:30 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi111-76716-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi111-76716-104"}]': finished 2024-09-16T09:10:31.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:30 smithi111 ceph-mon[29803]: osdmap e518: 8 total, 8 up, 8 in 2024-09-16T09:10:31.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:30 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi111-76716-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi111-76716-104"}]': finished 2024-09-16T09:10:31.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:30 smithi111 ceph-mon[40056]: osdmap e518: 8 total, 8 up, 8 in 2024-09-16T09:10:32.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:10:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:10:32.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:31 smithi114 ceph-mon[33315]: pgmap v788: 172 pgs: 8 unknown, 164 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:32.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:31 smithi114 ceph-mon[33315]: osdmap e519: 8 total, 8 up, 8 in 2024-09-16T09:10:32.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:31 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:32.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:32.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:31 smithi111 ceph-mon[29803]: pgmap v788: 172 pgs: 8 unknown, 164 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:32.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:31 smithi111 ceph-mon[29803]: osdmap e519: 8 total, 8 up, 8 in 2024-09-16T09:10:32.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:31 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:32.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:32.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:31 smithi111 ceph-mon[40056]: pgmap v788: 172 pgs: 8 unknown, 164 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:32.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:31 smithi111 ceph-mon[40056]: osdmap e519: 8 total, 8 up, 8 in 2024-09-16T09:10:32.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:31 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:32.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:32 smithi114 ceph-mon[33315]: osdmap e520: 8 total, 8 up, 8 in 2024-09-16T09:10:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:32 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:33.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:32 smithi114 ceph-mon[33315]: osdmap e521: 8 total, 8 up, 8 in 2024-09-16T09:10:33.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:10:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:10:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:10:33.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[29803]: osdmap e520: 8 total, 8 up, 8 in 2024-09-16T09:10:33.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:33.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:33.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:33.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:33.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[29803]: osdmap e521: 8 total, 8 up, 8 in 2024-09-16T09:10:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[40056]: osdmap e520: 8 total, 8 up, 8 in 2024-09-16T09:10:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:33.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:32 smithi111 ceph-mon[40056]: osdmap e521: 8 total, 8 up, 8 in 2024-09-16T09:10:34.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:33 smithi114 ceph-mon[33315]: pgmap v791: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:34.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:33 smithi114 ceph-mon[33315]: osdmap e522: 8 total, 8 up, 8 in 2024-09-16T09:10:34.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:34.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:33 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:33 smithi111 ceph-mon[29803]: pgmap v791: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:33 smithi111 ceph-mon[29803]: osdmap e522: 8 total, 8 up, 8 in 2024-09-16T09:10:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:34.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:33 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:34.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:33 smithi111 ceph-mon[40056]: pgmap v791: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:34.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:33 smithi111 ceph-mon[40056]: osdmap e522: 8 total, 8 up, 8 in 2024-09-16T09:10:34.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:34.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:33 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:36.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:35 smithi114 ceph-mon[33315]: pgmap v794: 236 pgs: 64 unknown, 172 active+clean; 583 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:36.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:36.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:35 smithi114 ceph-mon[33315]: osdmap e523: 8 total, 8 up, 8 in 2024-09-16T09:10:36.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:36.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:35 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:36.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[29803]: pgmap v794: 236 pgs: 64 unknown, 172 active+clean; 583 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:36.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:36.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[29803]: osdmap e523: 8 total, 8 up, 8 in 2024-09-16T09:10:36.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:36.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:36.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[40056]: pgmap v794: 236 pgs: 64 unknown, 172 active+clean; 583 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:10:36.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:36.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[40056]: osdmap e523: 8 total, 8 up, 8 in 2024-09-16T09:10:36.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:36.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:35 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:37.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"}]': finished 2024-09-16T09:10:37.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:36 smithi114 ceph-mon[33315]: osdmap e524: 8 total, 8 up, 8 in 2024-09-16T09:10:37.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-107", "overlaypool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:37.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-107", "overlaypool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:37.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:36 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-107", "overlaypool": "test-rados-api-smithi111-76716-107-cache"}]': finished 2024-09-16T09:10:37.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:36 smithi114 ceph-mon[33315]: osdmap e525: 8 total, 8 up, 8 in 2024-09-16T09:10:37.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"}]': finished 2024-09-16T09:10:37.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[29803]: osdmap e524: 8 total, 8 up, 8 in 2024-09-16T09:10:37.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-107", "overlaypool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:37.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-107", "overlaypool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:37.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-107", "overlaypool": "test-rados-api-smithi111-76716-107-cache"}]': finished 2024-09-16T09:10:37.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[29803]: osdmap e525: 8 total, 8 up, 8 in 2024-09-16T09:10:37.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"}]': finished 2024-09-16T09:10:37.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[40056]: osdmap e524: 8 total, 8 up, 8 in 2024-09-16T09:10:37.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-107", "overlaypool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:37.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-107", "overlaypool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:37.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-107", "overlaypool": "test-rados-api-smithi111-76716-107-cache"}]': finished 2024-09-16T09:10:37.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:36 smithi111 ceph-mon[40056]: osdmap e525: 8 total, 8 up, 8 in 2024-09-16T09:10:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:37 smithi114 ceph-mon[33315]: pgmap v797: 236 pgs: 236 active+clean; 583 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:37 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:37 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-107-cache", "mode": "writeback"} : dispatch 2024-09-16T09:10:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-107-cache", "mode": "writeback"} : dispatch 2024-09-16T09:10:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:37 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:37 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-107-cache", "mode": "writeback"}]': finished 2024-09-16T09:10:38.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:37 smithi114 ceph-mon[33315]: osdmap e526: 8 total, 8 up, 8 in 2024-09-16T09:10:38.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[40056]: pgmap v797: 236 pgs: 236 active+clean; 583 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:38.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:38.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-107-cache", "mode": "writeback"} : dispatch 2024-09-16T09:10:38.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-107-cache", "mode": "writeback"} : dispatch 2024-09-16T09:10:38.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:38.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-107-cache", "mode": "writeback"}]': finished 2024-09-16T09:10:38.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[40056]: osdmap e526: 8 total, 8 up, 8 in 2024-09-16T09:10:38.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[29803]: pgmap v797: 236 pgs: 236 active+clean; 583 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:38.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:38.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-107-cache", "mode": "writeback"} : dispatch 2024-09-16T09:10:38.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-107-cache", "mode": "writeback"} : dispatch 2024-09-16T09:10:38.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:38.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-107-cache", "mode": "writeback"}]': finished 2024-09-16T09:10:38.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:37 smithi111 ceph-mon[29803]: osdmap e526: 8 total, 8 up, 8 in 2024-09-16T09:10:39.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-107"} : dispatch 2024-09-16T09:10:39.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:38 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-107"} : dispatch 2024-09-16T09:10:39.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-107"} : dispatch 2024-09-16T09:10:39.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:38 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-107"} : dispatch 2024-09-16T09:10:39.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-107"} : dispatch 2024-09-16T09:10:39.428 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:38 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-107"} : dispatch 2024-09-16T09:10:40.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:39 smithi114 ceph-mon[33315]: pgmap v800: 236 pgs: 236 active+clean; 583 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:40.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:39 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-107"}]': finished 2024-09-16T09:10:40.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:39 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:40.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:39 smithi114 ceph-mon[33315]: osdmap e527: 8 total, 8 up, 8 in 2024-09-16T09:10:40.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:39 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[29803]: pgmap v800: 236 pgs: 236 active+clean; 583 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-107"}]': finished 2024-09-16T09:10:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[29803]: osdmap e527: 8 total, 8 up, 8 in 2024-09-16T09:10:40.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:40.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[40056]: pgmap v800: 236 pgs: 236 active+clean; 583 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:40.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-107"}]': finished 2024-09-16T09:10:40.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3732869884' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:40.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[40056]: osdmap e527: 8 total, 8 up, 8 in 2024-09-16T09:10:40.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:39 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"} : dispatch 2024-09-16T09:10:41.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:40 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:40 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"}]': finished 2024-09-16T09:10:41.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:40 smithi114 ceph-mon[33315]: osdmap e528: 8 total, 8 up, 8 in 2024-09-16T09:10:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:40 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:40 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"}]': finished 2024-09-16T09:10:41.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:40 smithi111 ceph-mon[29803]: osdmap e528: 8 total, 8 up, 8 in 2024-09-16T09:10:41.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:40 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:41.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:40 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-107", "tierpool": "test-rados-api-smithi111-76716-107-cache"}]': finished 2024-09-16T09:10:41.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:40 smithi111 ceph-mon[40056]: osdmap e528: 8 total, 8 up, 8 in 2024-09-16T09:10:42.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:41 smithi114 ceph-mon[33315]: pgmap v803: 236 pgs: 236 active+clean; 583 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:42.555 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:41 smithi114 ceph-mon[33315]: osdmap e529: 8 total, 8 up, 8 in 2024-09-16T09:10:42.556 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:10:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:10:42.556 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:41 smithi111 ceph-mon[29803]: pgmap v803: 236 pgs: 236 active+clean; 583 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:42.556 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:41 smithi111 ceph-mon[29803]: osdmap e529: 8 total, 8 up, 8 in 2024-09-16T09:10:42.557 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:41 smithi111 ceph-mon[40056]: pgmap v803: 236 pgs: 236 active+clean; 583 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:42.557 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:41 smithi111 ceph-mon[40056]: osdmap e529: 8 total, 8 up, 8 in 2024-09-16T09:10:43.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:42 smithi114 ceph-mon[33315]: osdmap e530: 8 total, 8 up, 8 in 2024-09-16T09:10:43.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:43.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:43.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:10:43.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:43.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:10:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:10:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:10:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[29803]: osdmap e530: 8 total, 8 up, 8 in 2024-09-16T09:10:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:43.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:10:43.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:43.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[40056]: osdmap e530: 8 total, 8 up, 8 in 2024-09-16T09:10:43.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:43.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:43.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:10:43.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:44.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:43 smithi114 ceph-mon[33315]: pgmap v806: 172 pgs: 172 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:44.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:10:44.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:43 smithi114 ceph-mon[33315]: osdmap e531: 8 total, 8 up, 8 in 2024-09-16T09:10:44.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:44.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:43 smithi111 ceph-mon[29803]: pgmap v806: 172 pgs: 172 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:44.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:10:44.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:43 smithi111 ceph-mon[29803]: osdmap e531: 8 total, 8 up, 8 in 2024-09-16T09:10:44.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:44.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:43 smithi111 ceph-mon[40056]: pgmap v806: 172 pgs: 172 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:44.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:10:44.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:43 smithi111 ceph-mon[40056]: osdmap e531: 8 total, 8 up, 8 in 2024-09-16T09:10:44.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:45.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:45 smithi114 ceph-mon[33315]: osdmap e532: 8 total, 8 up, 8 in 2024-09-16T09:10:45.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:10:45.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:45 smithi114 ceph-mon[33315]: osdmap e533: 8 total, 8 up, 8 in 2024-09-16T09:10:45.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:45 smithi111 ceph-mon[29803]: osdmap e532: 8 total, 8 up, 8 in 2024-09-16T09:10:45.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:10:45.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:45 smithi111 ceph-mon[29803]: osdmap e533: 8 total, 8 up, 8 in 2024-09-16T09:10:45.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:45 smithi111 ceph-mon[40056]: osdmap e532: 8 total, 8 up, 8 in 2024-09-16T09:10:45.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:10:45.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:45 smithi111 ceph-mon[40056]: osdmap e533: 8 total, 8 up, 8 in 2024-09-16T09:10:46.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:46 smithi114 ceph-mon[33315]: pgmap v809: 172 pgs: 172 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:46.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:46 smithi114 ceph-mon[33315]: osdmap e534: 8 total, 8 up, 8 in 2024-09-16T09:10:46.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-109-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:46 smithi111 ceph-mon[29803]: pgmap v809: 172 pgs: 172 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:46 smithi111 ceph-mon[29803]: osdmap e534: 8 total, 8 up, 8 in 2024-09-16T09:10:46.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-109-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:46 smithi111 ceph-mon[40056]: pgmap v809: 172 pgs: 172 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:46 smithi111 ceph-mon[40056]: osdmap e534: 8 total, 8 up, 8 in 2024-09-16T09:10:46.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-109-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:10:47.423 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:47.423 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:47.423 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:47 smithi114 ceph-mon[33315]: osdmap e535: 8 total, 8 up, 8 in 2024-09-16T09:10:47.424 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"} : dispatch 2024-09-16T09:10:47.429 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:47.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:47.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:47 smithi111 ceph-mon[29803]: osdmap e535: 8 total, 8 up, 8 in 2024-09-16T09:10:47.430 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"} : dispatch 2024-09-16T09:10:47.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:10:47.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:10:47.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:47 smithi111 ceph-mon[40056]: osdmap e535: 8 total, 8 up, 8 in 2024-09-16T09:10:47.430 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"} : dispatch 2024-09-16T09:10:48.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:48 smithi114 ceph-mon[33315]: pgmap v812: 212 pgs: 32 creating+peering, 180 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:48.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:48 smithi114 ceph-mon[33315]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:48.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"}]': finished 2024-09-16T09:10:48.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:48 smithi114 ceph-mon[33315]: osdmap e536: 8 total, 8 up, 8 in 2024-09-16T09:10:48.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-109", "overlaypool": "test-rados-api-smithi111-76716-109-cache"} : dispatch 2024-09-16T09:10:48.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[29803]: pgmap v812: 212 pgs: 32 creating+peering, 180 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:48.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[29803]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:48.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"}]': finished 2024-09-16T09:10:48.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[29803]: osdmap e536: 8 total, 8 up, 8 in 2024-09-16T09:10:48.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-109", "overlaypool": "test-rados-api-smithi111-76716-109-cache"} : dispatch 2024-09-16T09:10:48.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[40056]: pgmap v812: 212 pgs: 32 creating+peering, 180 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:48.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[40056]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:48.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"}]': finished 2024-09-16T09:10:48.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[40056]: osdmap e536: 8 total, 8 up, 8 in 2024-09-16T09:10:48.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-109", "overlaypool": "test-rados-api-smithi111-76716-109-cache"} : dispatch 2024-09-16T09:10:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:50 smithi114 ceph-mon[33315]: pgmap v815: 212 pgs: 32 creating+peering, 180 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-109", "overlaypool": "test-rados-api-smithi111-76716-109-cache"}]': finished 2024-09-16T09:10:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:50 smithi114 ceph-mon[33315]: osdmap e537: 8 total, 8 up, 8 in 2024-09-16T09:10:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-109-cache", "mode": "writeback"} : dispatch 2024-09-16T09:10:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:50 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:10:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:50 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:10:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:50 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:10:50.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:50 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:10:50.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[40056]: pgmap v815: 212 pgs: 32 creating+peering, 180 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:50.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-109", "overlaypool": "test-rados-api-smithi111-76716-109-cache"}]': finished 2024-09-16T09:10:50.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[40056]: osdmap e537: 8 total, 8 up, 8 in 2024-09-16T09:10:50.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-109-cache", "mode": "writeback"} : dispatch 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[29803]: pgmap v815: 212 pgs: 32 creating+peering, 180 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-109", "overlaypool": "test-rados-api-smithi111-76716-109-cache"}]': finished 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[29803]: osdmap e537: 8 total, 8 up, 8 in 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-109-cache", "mode": "writeback"} : dispatch 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:10:50.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:10:50.428 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:50 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:10:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:51 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-109-cache", "mode": "writeback"}]': finished 2024-09-16T09:10:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:51 smithi114 ceph-mon[33315]: osdmap e538: 8 total, 8 up, 8 in 2024-09-16T09:10:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:10:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:10:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:51 smithi114 ceph-mon[33315]: osdmap e539: 8 total, 8 up, 8 in 2024-09-16T09:10:51.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:51.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:51.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-109-cache", "mode": "writeback"}]': finished 2024-09-16T09:10:51.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[29803]: osdmap e538: 8 total, 8 up, 8 in 2024-09-16T09:10:51.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:10:51.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:10:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[29803]: osdmap e539: 8 total, 8 up, 8 in 2024-09-16T09:10:51.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:10:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-109-cache", "mode": "writeback"}]': finished 2024-09-16T09:10:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[40056]: osdmap e538: 8 total, 8 up, 8 in 2024-09-16T09:10:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:10:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:10:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[40056]: osdmap e539: 8 total, 8 up, 8 in 2024-09-16T09:10:51.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:10:52.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:10:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:10:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:52 smithi114 ceph-mon[33315]: pgmap v818: 212 pgs: 32 creating+peering, 180 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:10:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:52 smithi114 ceph-mon[33315]: osdmap e540: 8 total, 8 up, 8 in 2024-09-16T09:10:52.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:10:52.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:52 smithi111 ceph-mon[40056]: pgmap v818: 212 pgs: 32 creating+peering, 180 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:52.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:10:52.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:52 smithi111 ceph-mon[40056]: osdmap e540: 8 total, 8 up, 8 in 2024-09-16T09:10:52.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:10:52.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:52 smithi111 ceph-mon[29803]: pgmap v818: 212 pgs: 32 creating+peering, 180 active+clean; 583 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:10:52.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:10:52.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:52 smithi111 ceph-mon[29803]: osdmap e540: 8 total, 8 up, 8 in 2024-09-16T09:10:52.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:10:53.315 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:53.315 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:53 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:53.315 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:10:53.315 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:53 smithi114 ceph-mon[33315]: osdmap e541: 8 total, 8 up, 8 in 2024-09-16T09:10:53.315 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "min_read_recency_for_promote","val": "4"} : dispatch 2024-09-16T09:10:53.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:10:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[29803]: osdmap e541: 8 total, 8 up, 8 in 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "min_read_recency_for_promote","val": "4"} : dispatch 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[40056]: osdmap e541: 8 total, 8 up, 8 in 2024-09-16T09:10:53.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "min_read_recency_for_promote","val": "4"} : dispatch 2024-09-16T09:10:54.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:54 smithi114 ceph-mon[33315]: pgmap v821: 212 pgs: 212 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:54.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:54 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:54.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-09-16T09:10:54.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:54 smithi114 ceph-mon[33315]: osdmap e542: 8 total, 8 up, 8 in 2024-09-16T09:10:54.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:54 smithi111 ceph-mon[29803]: pgmap v821: 212 pgs: 212 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:54.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:54 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:54.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-09-16T09:10:54.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:54 smithi111 ceph-mon[29803]: osdmap e542: 8 total, 8 up, 8 in 2024-09-16T09:10:54.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:54 smithi111 ceph-mon[40056]: pgmap v821: 212 pgs: 212 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:54.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:54 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:54.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-09-16T09:10:54.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:54 smithi111 ceph-mon[40056]: osdmap e542: 8 total, 8 up, 8 in 2024-09-16T09:10:55.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:55.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:55.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:56.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:56 smithi114 ceph-mon[33315]: pgmap v824: 212 pgs: 212 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:56.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:10:56.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:56 smithi114 ceph-mon[33315]: osdmap e543: 8 total, 8 up, 8 in 2024-09-16T09:10:56.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:56 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"} : dispatch 2024-09-16T09:10:56.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:56 smithi111 ceph-mon[40056]: pgmap v824: 212 pgs: 212 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:56.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:10:56.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:56 smithi111 ceph-mon[40056]: osdmap e543: 8 total, 8 up, 8 in 2024-09-16T09:10:56.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:56 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"} : dispatch 2024-09-16T09:10:56.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:56 smithi111 ceph-mon[29803]: pgmap v824: 212 pgs: 212 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:10:56.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:10:56.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:56 smithi111 ceph-mon[29803]: osdmap e543: 8 total, 8 up, 8 in 2024-09-16T09:10:56.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:56 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"} : dispatch 2024-09-16T09:10:57.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"}]': finished 2024-09-16T09:10:57.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:57 smithi114 ceph-mon[33315]: osdmap e544: 8 total, 8 up, 8 in 2024-09-16T09:10:57.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"}]': finished 2024-09-16T09:10:57.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:57 smithi111 ceph-mon[40056]: osdmap e544: 8 total, 8 up, 8 in 2024-09-16T09:10:57.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-109", "tierpool": "test-rados-api-smithi111-76716-109-cache"}]': finished 2024-09-16T09:10:57.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:57 smithi111 ceph-mon[29803]: osdmap e544: 8 total, 8 up, 8 in 2024-09-16T09:10:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:58 smithi114 ceph-mon[33315]: pgmap v827: 212 pgs: 212 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:10:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:58 smithi114 ceph-mon[33315]: osdmap e545: 8 total, 8 up, 8 in 2024-09-16T09:10:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:58 smithi114 ceph-mon[33315]: osdmap e546: 8 total, 8 up, 8 in 2024-09-16T09:10:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:58.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:58 smithi111 ceph-mon[29803]: pgmap v827: 212 pgs: 212 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:10:58.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:58 smithi111 ceph-mon[29803]: osdmap e545: 8 total, 8 up, 8 in 2024-09-16T09:10:58.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:58 smithi111 ceph-mon[29803]: osdmap e546: 8 total, 8 up, 8 in 2024-09-16T09:10:58.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:58 smithi111 ceph-mon[40056]: pgmap v827: 212 pgs: 212 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:10:58.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:58 smithi111 ceph-mon[40056]: osdmap e545: 8 total, 8 up, 8 in 2024-09-16T09:10:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:58 smithi111 ceph-mon[40056]: osdmap e546: 8 total, 8 up, 8 in 2024-09-16T09:10:58.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:10:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:59 smithi114 ceph-mon[33315]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:10:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:10:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:59 smithi111 ceph-mon[29803]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:59.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:10:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:10:59.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:59 smithi111 ceph-mon[40056]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:10:59.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:10:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:11:00.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:00 smithi114 ceph-mon[33315]: pgmap v830: 172 pgs: 172 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:00 smithi114 ceph-mon[33315]: osdmap e547: 8 total, 8 up, 8 in 2024-09-16T09:11:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:11:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:11:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:00 smithi114 ceph-mon[33315]: osdmap e548: 8 total, 8 up, 8 in 2024-09-16T09:11:00.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[40056]: pgmap v830: 172 pgs: 172 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:00.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[40056]: osdmap e547: 8 total, 8 up, 8 in 2024-09-16T09:11:00.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:11:00.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:11:00.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[40056]: osdmap e548: 8 total, 8 up, 8 in 2024-09-16T09:11:00.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[29803]: pgmap v830: 172 pgs: 172 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:00.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[29803]: osdmap e547: 8 total, 8 up, 8 in 2024-09-16T09:11:00.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-109"} : dispatch 2024-09-16T09:11:00.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2087945554' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-109"}]': finished 2024-09-16T09:11:00.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:00 smithi111 ceph-mon[29803]: osdmap e548: 8 total, 8 up, 8 in 2024-09-16T09:11:02.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:11:02 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:11:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:02 smithi114 ceph-mon[33315]: pgmap v833: 172 pgs: 172 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:02 smithi114 ceph-mon[33315]: osdmap e549: 8 total, 8 up, 8 in 2024-09-16T09:11:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:02.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:02.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:02 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[29803]: pgmap v833: 172 pgs: 172 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[29803]: osdmap e549: 8 total, 8 up, 8 in 2024-09-16T09:11:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:02.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[40056]: pgmap v833: 172 pgs: 172 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[40056]: osdmap e549: 8 total, 8 up, 8 in 2024-09-16T09:11:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:02 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"}]': finished 2024-09-16T09:11:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:03 smithi114 ceph-mon[33315]: osdmap e550: 8 total, 8 up, 8 in 2024-09-16T09:11:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:03 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:03.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:11:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:11:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"}]': finished 2024-09-16T09:11:03.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[29803]: osdmap e550: 8 total, 8 up, 8 in 2024-09-16T09:11:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:03.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi111-76716-104"}]': finished 2024-09-16T09:11:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[40056]: osdmap e550: 8 total, 8 up, 8 in 2024-09-16T09:11:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3672528593' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"} : dispatch 2024-09-16T09:11:03.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:03 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:04 smithi114 ceph-mon[33315]: pgmap v836: 164 pgs: 164 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-16T09:11:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"}]': finished 2024-09-16T09:11:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:04 smithi114 ceph-mon[33315]: osdmap e551: 8 total, 8 up, 8 in 2024-09-16T09:11:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:04 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:11:04.415 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:04 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[29803]: pgmap v836: 164 pgs: 164 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-16T09:11:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"}]': finished 2024-09-16T09:11:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[29803]: osdmap e551: 8 total, 8 up, 8 in 2024-09-16T09:11:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:04.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:11:04.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:04.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[40056]: pgmap v836: 164 pgs: 164 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-16T09:11:04.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi111-76716-104"}]': finished 2024-09-16T09:11:04.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[40056]: osdmap e551: 8 total, 8 up, 8 in 2024-09-16T09:11:04.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:04.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:04.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-16T09:11:04.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:04 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:05.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:11:05.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:05 smithi114 ceph-mon[33315]: osdmap e552: 8 total, 8 up, 8 in 2024-09-16T09:11:05.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:05.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:11:05.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:05 smithi111 ceph-mon[29803]: osdmap e552: 8 total, 8 up, 8 in 2024-09-16T09:11:05.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:05.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi111-76716-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T09:11:05.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:05 smithi111 ceph-mon[40056]: osdmap e552: 8 total, 8 up, 8 in 2024-09-16T09:11:05.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:06.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:06 smithi114 ceph-mon[33315]: pgmap v839: 164 pgs: 164 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-16T09:11:06.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:06 smithi114 ceph-mon[33315]: osdmap e553: 8 total, 8 up, 8 in 2024-09-16T09:11:06.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:06 smithi111 ceph-mon[29803]: pgmap v839: 164 pgs: 164 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-16T09:11:06.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:06 smithi111 ceph-mon[29803]: osdmap e553: 8 total, 8 up, 8 in 2024-09-16T09:11:06.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:06 smithi111 ceph-mon[40056]: pgmap v839: 164 pgs: 164 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-16T09:11:06.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:06 smithi111 ceph-mon[40056]: osdmap e553: 8 total, 8 up, 8 in 2024-09-16T09:11:07.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:07.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:07 smithi114 ceph-mon[33315]: osdmap e554: 8 total, 8 up, 8 in 2024-09-16T09:11:07.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:07.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:07 smithi111 ceph-mon[29803]: osdmap e554: 8 total, 8 up, 8 in 2024-09-16T09:11:07.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi111-76716-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:07.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:07 smithi111 ceph-mon[40056]: osdmap e554: 8 total, 8 up, 8 in 2024-09-16T09:11:08.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:08 smithi114 ceph-mon[33315]: pgmap v842: 172 pgs: 8 creating+peering, 164 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:08.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:08 smithi114 ceph-mon[33315]: osdmap e555: 8 total, 8 up, 8 in 2024-09-16T09:11:08.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-112","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:08.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:08 smithi111 ceph-mon[29803]: pgmap v842: 172 pgs: 8 creating+peering, 164 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:08.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:08 smithi111 ceph-mon[29803]: osdmap e555: 8 total, 8 up, 8 in 2024-09-16T09:11:08.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-112","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:08.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:08 smithi111 ceph-mon[40056]: pgmap v842: 172 pgs: 8 creating+peering, 164 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:08.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:08 smithi111 ceph-mon[40056]: osdmap e555: 8 total, 8 up, 8 in 2024-09-16T09:11:08.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-112","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:09.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:09.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:09 smithi114 ceph-mon[33315]: osdmap e556: 8 total, 8 up, 8 in 2024-09-16T09:11:09.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:09.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:09 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:09.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:09.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:09 smithi111 ceph-mon[29803]: osdmap e556: 8 total, 8 up, 8 in 2024-09-16T09:11:09.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:09.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:09 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:09.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:09.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:09 smithi111 ceph-mon[40056]: osdmap e556: 8 total, 8 up, 8 in 2024-09-16T09:11:09.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:09.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:09 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:10.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:10 smithi114 ceph-mon[33315]: pgmap v845: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:10.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:10.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:10 smithi114 ceph-mon[33315]: osdmap e557: 8 total, 8 up, 8 in 2024-09-16T09:11:10.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-112"} : dispatch 2024-09-16T09:11:10.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:10 smithi111 ceph-mon[29803]: pgmap v845: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:10.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:10.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:10 smithi111 ceph-mon[29803]: osdmap e557: 8 total, 8 up, 8 in 2024-09-16T09:11:10.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-112"} : dispatch 2024-09-16T09:11:10.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:10 smithi111 ceph-mon[40056]: pgmap v845: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:10.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:10.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:10 smithi111 ceph-mon[40056]: osdmap e557: 8 total, 8 up, 8 in 2024-09-16T09:11:10.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-112"} : dispatch 2024-09-16T09:11:11.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-112"}]': finished 2024-09-16T09:11:11.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:11 smithi114 ceph-mon[33315]: osdmap e558: 8 total, 8 up, 8 in 2024-09-16T09:11:11.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:11.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-112"}]': finished 2024-09-16T09:11:11.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:11 smithi111 ceph-mon[29803]: osdmap e558: 8 total, 8 up, 8 in 2024-09-16T09:11:11.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:11.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-112"}]': finished 2024-09-16T09:11:11.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:11 smithi111 ceph-mon[40056]: osdmap e558: 8 total, 8 up, 8 in 2024-09-16T09:11:11.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:12.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:11:12 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:11:12.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:12 smithi114 ceph-mon[33315]: pgmap v848: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:12.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:12.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:12 smithi114 ceph-mon[33315]: osdmap e559: 8 total, 8 up, 8 in 2024-09-16T09:11:12.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112"} : dispatch 2024-09-16T09:11:12.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:12 smithi111 ceph-mon[29803]: pgmap v848: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:12.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:12.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:12 smithi111 ceph-mon[29803]: osdmap e559: 8 total, 8 up, 8 in 2024-09-16T09:11:12.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112"} : dispatch 2024-09-16T09:11:12.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:12 smithi111 ceph-mon[40056]: pgmap v848: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:12.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:12.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:12 smithi111 ceph-mon[40056]: osdmap e559: 8 total, 8 up, 8 in 2024-09-16T09:11:12.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112"} : dispatch 2024-09-16T09:11:13.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112"}]': finished 2024-09-16T09:11:13.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:13.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:13 smithi114 ceph-mon[33315]: osdmap e560: 8 total, 8 up, 8 in 2024-09-16T09:11:13.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:11:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:11:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:11:13.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112"}]': finished 2024-09-16T09:11:13.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:13.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:13 smithi111 ceph-mon[29803]: osdmap e560: 8 total, 8 up, 8 in 2024-09-16T09:11:13.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-112"}]': finished 2024-09-16T09:11:13.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:13.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:13 smithi111 ceph-mon[40056]: osdmap e560: 8 total, 8 up, 8 in 2024-09-16T09:11:14.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:14 smithi114 ceph-mon[33315]: pgmap v851: 204 pgs: 204 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:11:14.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:14 smithi114 ceph-mon[33315]: osdmap e561: 8 total, 8 up, 8 in 2024-09-16T09:11:14.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:14 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:14.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:14 smithi111 ceph-mon[29803]: pgmap v851: 204 pgs: 204 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:11:14.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:14 smithi111 ceph-mon[29803]: osdmap e561: 8 total, 8 up, 8 in 2024-09-16T09:11:14.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:14 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:14.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:14 smithi111 ceph-mon[40056]: pgmap v851: 204 pgs: 204 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-16T09:11:14.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:14 smithi111 ceph-mon[40056]: osdmap e561: 8 total, 8 up, 8 in 2024-09-16T09:11:14.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:14 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:15.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:15 smithi114 ceph-mon[33315]: osdmap e562: 8 total, 8 up, 8 in 2024-09-16T09:11:15.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-114","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:15.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:15 smithi111 ceph-mon[29803]: osdmap e562: 8 total, 8 up, 8 in 2024-09-16T09:11:15.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-114","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:15.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:15 smithi111 ceph-mon[40056]: osdmap e562: 8 total, 8 up, 8 in 2024-09-16T09:11:15.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-114","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:16.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:16 smithi114 ceph-mon[33315]: pgmap v854: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:11:16.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:16.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:16 smithi114 ceph-mon[33315]: osdmap e563: 8 total, 8 up, 8 in 2024-09-16T09:11:16.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:16.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:16 smithi111 ceph-mon[29803]: pgmap v854: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:11:16.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:16.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:16 smithi111 ceph-mon[29803]: osdmap e563: 8 total, 8 up, 8 in 2024-09-16T09:11:16.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:16.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:16 smithi111 ceph-mon[40056]: pgmap v854: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:11:16.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:16.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:16 smithi111 ceph-mon[40056]: osdmap e563: 8 total, 8 up, 8 in 2024-09-16T09:11:16.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:17.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:17.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:17 smithi111 ceph-mon[29803]: osdmap e564: 8 total, 8 up, 8 in 2024-09-16T09:11:17.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-114"} : dispatch 2024-09-16T09:11:17.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:17.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:17.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:17 smithi111 ceph-mon[40056]: osdmap e564: 8 total, 8 up, 8 in 2024-09-16T09:11:17.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-114"} : dispatch 2024-09-16T09:11:17.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:17 smithi114 ceph-mon[33315]: osdmap e564: 8 total, 8 up, 8 in 2024-09-16T09:11:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-114"} : dispatch 2024-09-16T09:11:17.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:18.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:18 smithi111 ceph-mon[29803]: pgmap v857: 204 pgs: 204 active+clean; 583 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-16T09:11:18.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-114"}]': finished 2024-09-16T09:11:18.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:18 smithi111 ceph-mon[29803]: osdmap e565: 8 total, 8 up, 8 in 2024-09-16T09:11:18.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-114", "mode": "writeback"} : dispatch 2024-09-16T09:11:18.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:18 smithi111 ceph-mon[40056]: pgmap v857: 204 pgs: 204 active+clean; 583 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-16T09:11:18.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-114"}]': finished 2024-09-16T09:11:18.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:18 smithi111 ceph-mon[40056]: osdmap e565: 8 total, 8 up, 8 in 2024-09-16T09:11:18.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-114", "mode": "writeback"} : dispatch 2024-09-16T09:11:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:18 smithi114 ceph-mon[33315]: pgmap v857: 204 pgs: 204 active+clean; 583 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-16T09:11:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-114"}]': finished 2024-09-16T09:11:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:18 smithi114 ceph-mon[33315]: osdmap e565: 8 total, 8 up, 8 in 2024-09-16T09:11:18.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-114", "mode": "writeback"} : dispatch 2024-09-16T09:11:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:19 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:11:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-114", "mode": "writeback"}]': finished 2024-09-16T09:11:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:19 smithi114 ceph-mon[33315]: osdmap e566: 8 total, 8 up, 8 in 2024-09-16T09:11:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:19.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:19 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:11:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-114", "mode": "writeback"}]': finished 2024-09-16T09:11:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[40056]: osdmap e566: 8 total, 8 up, 8 in 2024-09-16T09:11:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:19.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:11:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-114", "mode": "writeback"}]': finished 2024-09-16T09:11:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[29803]: osdmap e566: 8 total, 8 up, 8 in 2024-09-16T09:11:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:19 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:20.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:20 smithi114 ceph-mon[33315]: pgmap v860: 204 pgs: 204 active+clean; 583 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:11:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:20 smithi114 ceph-mon[33315]: osdmap e567: 8 total, 8 up, 8 in 2024-09-16T09:11:20.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114"} : dispatch 2024-09-16T09:11:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:20 smithi111 ceph-mon[40056]: pgmap v860: 204 pgs: 204 active+clean; 583 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:11:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:20 smithi111 ceph-mon[40056]: osdmap e567: 8 total, 8 up, 8 in 2024-09-16T09:11:20.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114"} : dispatch 2024-09-16T09:11:20.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:20 smithi111 ceph-mon[29803]: pgmap v860: 204 pgs: 204 active+clean; 583 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:11:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:20 smithi111 ceph-mon[29803]: osdmap e567: 8 total, 8 up, 8 in 2024-09-16T09:11:20.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114"} : dispatch 2024-09-16T09:11:21.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:21 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:11:21.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114"}]': finished 2024-09-16T09:11:21.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:21 smithi114 ceph-mon[33315]: osdmap e568: 8 total, 8 up, 8 in 2024-09-16T09:11:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:21 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:11:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114"}]': finished 2024-09-16T09:11:21.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:21 smithi111 ceph-mon[29803]: osdmap e568: 8 total, 8 up, 8 in 2024-09-16T09:11:21.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:21 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:11:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-114"}]': finished 2024-09-16T09:11:21.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:21 smithi111 ceph-mon[40056]: osdmap e568: 8 total, 8 up, 8 in 2024-09-16T09:11:22.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:11:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:11:22.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:22 smithi114 ceph-mon[33315]: pgmap v863: 204 pgs: 204 active+clean; 583 KiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:22.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:22 smithi114 ceph-mon[33315]: osdmap e569: 8 total, 8 up, 8 in 2024-09-16T09:11:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:22 smithi111 ceph-mon[29803]: pgmap v863: 204 pgs: 204 active+clean; 583 KiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:22.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:22 smithi111 ceph-mon[29803]: osdmap e569: 8 total, 8 up, 8 in 2024-09-16T09:11:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:22 smithi111 ceph-mon[40056]: pgmap v863: 204 pgs: 204 active+clean; 583 KiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:22.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:22 smithi111 ceph-mon[40056]: osdmap e569: 8 total, 8 up, 8 in 2024-09-16T09:11:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:23 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:23 smithi114 ceph-mon[33315]: osdmap e570: 8 total, 8 up, 8 in 2024-09-16T09:11:23.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-116","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:23 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:23 smithi111 ceph-mon[29803]: osdmap e570: 8 total, 8 up, 8 in 2024-09-16T09:11:23.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-116","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:23.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:11:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:11:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:11:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:23 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:23 smithi111 ceph-mon[40056]: osdmap e570: 8 total, 8 up, 8 in 2024-09-16T09:11:23.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-116","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:24.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:24 smithi114 ceph-mon[33315]: pgmap v866: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:24 smithi114 ceph-mon[33315]: osdmap e571: 8 total, 8 up, 8 in 2024-09-16T09:11:24.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:24 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:24 smithi111 ceph-mon[40056]: pgmap v866: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:24 smithi111 ceph-mon[40056]: osdmap e571: 8 total, 8 up, 8 in 2024-09-16T09:11:24.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:24 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:24 smithi111 ceph-mon[29803]: pgmap v866: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:24.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:24.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:24 smithi111 ceph-mon[29803]: osdmap e571: 8 total, 8 up, 8 in 2024-09-16T09:11:24.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:24 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:25.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:25 smithi114 ceph-mon[33315]: osdmap e572: 8 total, 8 up, 8 in 2024-09-16T09:11:25.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:25 smithi111 ceph-mon[29803]: osdmap e572: 8 total, 8 up, 8 in 2024-09-16T09:11:25.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:25.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:25 smithi111 ceph-mon[40056]: osdmap e572: 8 total, 8 up, 8 in 2024-09-16T09:11:25.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:26 smithi114 ceph-mon[33315]: pgmap v869: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:26 smithi114 ceph-mon[33315]: osdmap e573: 8 total, 8 up, 8 in 2024-09-16T09:11:26.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-116"} : dispatch 2024-09-16T09:11:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:26 smithi111 ceph-mon[29803]: pgmap v869: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:26 smithi111 ceph-mon[29803]: osdmap e573: 8 total, 8 up, 8 in 2024-09-16T09:11:26.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-116"} : dispatch 2024-09-16T09:11:26.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:26 smithi111 ceph-mon[40056]: pgmap v869: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:26.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:26 smithi111 ceph-mon[40056]: osdmap e573: 8 total, 8 up, 8 in 2024-09-16T09:11:26.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-116"} : dispatch 2024-09-16T09:11:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-116"}]': finished 2024-09-16T09:11:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:27 smithi114 ceph-mon[33315]: osdmap e574: 8 total, 8 up, 8 in 2024-09-16T09:11:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-116", "mode": "writeback"} : dispatch 2024-09-16T09:11:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:27 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:11:27.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-116", "mode": "writeback"}]': finished 2024-09-16T09:11:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-116"}]': finished 2024-09-16T09:11:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[29803]: osdmap e574: 8 total, 8 up, 8 in 2024-09-16T09:11:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-116", "mode": "writeback"} : dispatch 2024-09-16T09:11:27.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:11:27.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-116", "mode": "writeback"}]': finished 2024-09-16T09:11:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-116"}]': finished 2024-09-16T09:11:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[40056]: osdmap e574: 8 total, 8 up, 8 in 2024-09-16T09:11:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-116", "mode": "writeback"} : dispatch 2024-09-16T09:11:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:11:27.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-116", "mode": "writeback"}]': finished 2024-09-16T09:11:28.242 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[29803]: pgmap v872: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-16T09:11:28.242 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[29803]: osdmap e575: 8 total, 8 up, 8 in 2024-09-16T09:11:28.242 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "325.5"} : dispatch 2024-09-16T09:11:28.242 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[29803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "325.5"}]: dispatch 2024-09-16T09:11:28.242 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[29803]: 325.5 scrub starts 2024-09-16T09:11:28.243 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[29803]: 325.5 scrub ok 2024-09-16T09:11:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:28 smithi114 ceph-mon[33315]: pgmap v872: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-16T09:11:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:28 smithi114 ceph-mon[33315]: osdmap e575: 8 total, 8 up, 8 in 2024-09-16T09:11:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "325.5"} : dispatch 2024-09-16T09:11:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:28 smithi114 ceph-mon[33315]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "325.5"}]: dispatch 2024-09-16T09:11:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:28 smithi114 ceph-mon[33315]: 325.5 scrub starts 2024-09-16T09:11:28.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:28 smithi114 ceph-mon[33315]: 325.5 scrub ok 2024-09-16T09:11:28.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[40056]: pgmap v872: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-16T09:11:28.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[40056]: osdmap e575: 8 total, 8 up, 8 in 2024-09-16T09:11:28.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "pg scrub", "pgid": "325.5"} : dispatch 2024-09-16T09:11:28.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[40056]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "325.5"}]: dispatch 2024-09-16T09:11:28.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[40056]: 325.5 scrub starts 2024-09-16T09:11:28.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:28 smithi111 ceph-mon[40056]: 325.5 scrub ok 2024-09-16T09:11:29.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:29 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:29.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:29 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:29.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:29 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:30.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:30 smithi114 ceph-mon[33315]: pgmap v874: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-16T09:11:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:30 smithi111 ceph-mon[29803]: pgmap v874: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-16T09:11:30.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:30 smithi111 ceph-mon[40056]: pgmap v874: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-16T09:11:32.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:11:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:11:32.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:32 smithi114 ceph-mon[33315]: pgmap v875: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:11:32.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:32.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:32 smithi111 ceph-mon[40056]: pgmap v875: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:11:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:32.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:32 smithi111 ceph-mon[29803]: pgmap v875: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:11:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:33 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:33 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:11:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:11:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:11:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:33 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:34.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:34 smithi114 ceph-mon[33315]: pgmap v876: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 961 B/s wr, 2 op/s 2024-09-16T09:11:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:34 smithi111 ceph-mon[29803]: pgmap v876: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 961 B/s wr, 2 op/s 2024-09-16T09:11:34.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:34 smithi111 ceph-mon[40056]: pgmap v876: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 961 B/s wr, 2 op/s 2024-09-16T09:11:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:36 smithi114 ceph-mon[33315]: pgmap v877: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 848 B/s wr, 2 op/s 2024-09-16T09:11:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:36 smithi111 ceph-mon[29803]: pgmap v877: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 848 B/s wr, 2 op/s 2024-09-16T09:11:36.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:36 smithi111 ceph-mon[40056]: pgmap v877: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 848 B/s wr, 2 op/s 2024-09-16T09:11:38.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:38 smithi114 ceph-mon[33315]: pgmap v878: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:38 smithi111 ceph-mon[29803]: pgmap v878: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:38.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:38 smithi111 ceph-mon[40056]: pgmap v878: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:40 smithi114 ceph-mon[33315]: pgmap v879: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T09:11:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:40 smithi111 ceph-mon[29803]: pgmap v879: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T09:11:40.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:40 smithi111 ceph-mon[40056]: pgmap v879: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T09:11:42.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:42 smithi114 ceph-mon[33315]: pgmap v880: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-16T09:11:42.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:11:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:11:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:42 smithi111 ceph-mon[29803]: pgmap v880: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-16T09:11:42.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:42 smithi111 ceph-mon[40056]: pgmap v880: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-16T09:11:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:43 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:43 smithi114 ceph-mon[33315]: osdmap e576: 8 total, 8 up, 8 in 2024-09-16T09:11:43.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:43 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:43 smithi111 ceph-mon[29803]: osdmap e576: 8 total, 8 up, 8 in 2024-09-16T09:11:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:43.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:11:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:11:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:11:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:43 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:43 smithi111 ceph-mon[40056]: osdmap e576: 8 total, 8 up, 8 in 2024-09-16T09:11:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:44 smithi114 ceph-mon[33315]: pgmap v882: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:11:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:44 smithi114 ceph-mon[33315]: osdmap e577: 8 total, 8 up, 8 in 2024-09-16T09:11:44.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116"} : dispatch 2024-09-16T09:11:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:44 smithi111 ceph-mon[29803]: pgmap v882: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:11:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:44 smithi111 ceph-mon[29803]: osdmap e577: 8 total, 8 up, 8 in 2024-09-16T09:11:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116"} : dispatch 2024-09-16T09:11:44.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:44 smithi111 ceph-mon[40056]: pgmap v882: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:11:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:44 smithi111 ceph-mon[40056]: osdmap e577: 8 total, 8 up, 8 in 2024-09-16T09:11:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116"} : dispatch 2024-09-16T09:11:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:45 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:11:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116"}]': finished 2024-09-16T09:11:45.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:45 smithi114 ceph-mon[33315]: osdmap e578: 8 total, 8 up, 8 in 2024-09-16T09:11:45.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:45 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:11:45.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116"}]': finished 2024-09-16T09:11:45.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:45 smithi111 ceph-mon[29803]: osdmap e578: 8 total, 8 up, 8 in 2024-09-16T09:11:45.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:45 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:11:45.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-116"}]': finished 2024-09-16T09:11:45.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:45 smithi111 ceph-mon[40056]: osdmap e578: 8 total, 8 up, 8 in 2024-09-16T09:11:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:46 smithi114 ceph-mon[33315]: pgmap v885: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:11:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:46 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:46.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:46 smithi114 ceph-mon[33315]: osdmap e579: 8 total, 8 up, 8 in 2024-09-16T09:11:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:46 smithi111 ceph-mon[29803]: pgmap v885: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:11:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:46 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:46.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:46 smithi111 ceph-mon[29803]: osdmap e579: 8 total, 8 up, 8 in 2024-09-16T09:11:46.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:46 smithi111 ceph-mon[40056]: pgmap v885: 204 pgs: 204 active+clean; 583 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:11:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:46 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:46.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:46 smithi111 ceph-mon[40056]: osdmap e579: 8 total, 8 up, 8 in 2024-09-16T09:11:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:47 smithi114 ceph-mon[33315]: osdmap e580: 8 total, 8 up, 8 in 2024-09-16T09:11:47.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-118","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:47.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:47.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:47 smithi111 ceph-mon[29803]: osdmap e580: 8 total, 8 up, 8 in 2024-09-16T09:11:47.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-118","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:47.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:47.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:47 smithi111 ceph-mon[40056]: osdmap e580: 8 total, 8 up, 8 in 2024-09-16T09:11:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-118","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:47.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:11:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:48 smithi114 ceph-mon[33315]: pgmap v888: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-16T09:11:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:48 smithi114 ceph-mon[33315]: osdmap e581: 8 total, 8 up, 8 in 2024-09-16T09:11:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:48 smithi114 ceph-mon[33315]: osdmap e582: 8 total, 8 up, 8 in 2024-09-16T09:11:48.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[29803]: pgmap v888: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-16T09:11:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[29803]: osdmap e581: 8 total, 8 up, 8 in 2024-09-16T09:11:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[29803]: osdmap e582: 8 total, 8 up, 8 in 2024-09-16T09:11:48.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[40056]: pgmap v888: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-16T09:11:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[40056]: osdmap e581: 8 total, 8 up, 8 in 2024-09-16T09:11:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[40056]: osdmap e582: 8 total, 8 up, 8 in 2024-09-16T09:11:48.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[29803]: pgmap v891: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[29803]: osdmap e583: 8 total, 8 up, 8 in 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-118"} : dispatch 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[40056]: pgmap v891: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[40056]: osdmap e583: 8 total, 8 up, 8 in 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-118"} : dispatch 2024-09-16T09:11:50.614 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:50 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:11:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:50 smithi114 ceph-mon[33315]: pgmap v891: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-16T09:11:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:50 smithi114 ceph-mon[33315]: osdmap e583: 8 total, 8 up, 8 in 2024-09-16T09:11:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-118"} : dispatch 2024-09-16T09:11:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:50 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:11:51.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-118"}]': finished 2024-09-16T09:11:51.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:51 smithi114 ceph-mon[33315]: osdmap e584: 8 total, 8 up, 8 in 2024-09-16T09:11:51.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-118", "mode": "writeback"} : dispatch 2024-09-16T09:11:51.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:51 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:11:51.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-118", "mode": "writeback"}]': finished 2024-09-16T09:11:51.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:51 smithi114 ceph-mon[33315]: osdmap e585: 8 total, 8 up, 8 in 2024-09-16T09:11:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-118"}]': finished 2024-09-16T09:11:51.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[29803]: osdmap e584: 8 total, 8 up, 8 in 2024-09-16T09:11:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-118", "mode": "writeback"} : dispatch 2024-09-16T09:11:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:11:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-118", "mode": "writeback"}]': finished 2024-09-16T09:11:51.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[29803]: osdmap e585: 8 total, 8 up, 8 in 2024-09-16T09:11:51.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-118"}]': finished 2024-09-16T09:11:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[40056]: osdmap e584: 8 total, 8 up, 8 in 2024-09-16T09:11:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-118", "mode": "writeback"} : dispatch 2024-09-16T09:11:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:11:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-118", "mode": "writeback"}]': finished 2024-09-16T09:11:51.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:51 smithi111 ceph-mon[40056]: osdmap e585: 8 total, 8 up, 8 in 2024-09-16T09:11:52.370 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:52 smithi114 ceph-mon[33315]: pgmap v894: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:52.371 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:52 smithi114 ceph-mon[33315]: osdmap e586: 8 total, 8 up, 8 in 2024-09-16T09:11:52.371 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:11:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:11:52.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:52 smithi111 ceph-mon[40056]: pgmap v894: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:52.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:52 smithi111 ceph-mon[40056]: osdmap e586: 8 total, 8 up, 8 in 2024-09-16T09:11:52.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:52 smithi111 ceph-mon[29803]: pgmap v894: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:11:52.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:52 smithi111 ceph-mon[29803]: osdmap e586: 8 total, 8 up, 8 in 2024-09-16T09:11:53.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:53 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:53.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:53 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:11:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:11:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:53.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:53.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:11:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:11:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:53.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:53 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:54 smithi114 ceph-mon[33315]: pgmap v897: 204 pgs: 204 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:11:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:54 smithi114 ceph-mon[33315]: osdmap e587: 8 total, 8 up, 8 in 2024-09-16T09:11:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118"} : dispatch 2024-09-16T09:11:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:54 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:54 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:54 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:11:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:54 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:11:54.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:54 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[29803]: pgmap v897: 204 pgs: 204 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:11:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[29803]: osdmap e587: 8 total, 8 up, 8 in 2024-09-16T09:11:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118"} : dispatch 2024-09-16T09:11:54.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[40056]: pgmap v897: 204 pgs: 204 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[40056]: osdmap e587: 8 total, 8 up, 8 in 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118"} : dispatch 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:54.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:54.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:11:54.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:11:54.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:54 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:11:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:55 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:11:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118"}]': finished 2024-09-16T09:11:55.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:55 smithi114 ceph-mon[33315]: osdmap e588: 8 total, 8 up, 8 in 2024-09-16T09:11:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:55 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:11:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118"}]': finished 2024-09-16T09:11:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:55 smithi111 ceph-mon[29803]: osdmap e588: 8 total, 8 up, 8 in 2024-09-16T09:11:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:55 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:11:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-118"}]': finished 2024-09-16T09:11:55.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:55 smithi111 ceph-mon[40056]: osdmap e588: 8 total, 8 up, 8 in 2024-09-16T09:11:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:56 smithi114 ceph-mon[33315]: pgmap v900: 204 pgs: 204 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:11:56.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:56 smithi114 ceph-mon[33315]: osdmap e589: 8 total, 8 up, 8 in 2024-09-16T09:11:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:56 smithi111 ceph-mon[29803]: pgmap v900: 204 pgs: 204 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:11:56.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:56 smithi111 ceph-mon[29803]: osdmap e589: 8 total, 8 up, 8 in 2024-09-16T09:11:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:56 smithi111 ceph-mon[40056]: pgmap v900: 204 pgs: 204 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:11:56.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:56 smithi111 ceph-mon[40056]: osdmap e589: 8 total, 8 up, 8 in 2024-09-16T09:11:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:57 smithi114 ceph-mon[33315]: osdmap e590: 8 total, 8 up, 8 in 2024-09-16T09:11:57.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-120","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:57 smithi111 ceph-mon[29803]: osdmap e590: 8 total, 8 up, 8 in 2024-09-16T09:11:57.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-120","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:57.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:57 smithi111 ceph-mon[40056]: osdmap e590: 8 total, 8 up, 8 in 2024-09-16T09:11:57.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-120","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:11:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:58 smithi114 ceph-mon[33315]: pgmap v903: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:58 smithi114 ceph-mon[33315]: osdmap e591: 8 total, 8 up, 8 in 2024-09-16T09:11:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:58 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:58 smithi114 ceph-mon[33315]: osdmap e592: 8 total, 8 up, 8 in 2024-09-16T09:11:58.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:11:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-120"} : dispatch 2024-09-16T09:11:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[29803]: pgmap v903: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:58.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[29803]: osdmap e591: 8 total, 8 up, 8 in 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[29803]: osdmap e592: 8 total, 8 up, 8 in 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-120"} : dispatch 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[40056]: pgmap v903: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[40056]: osdmap e591: 8 total, 8 up, 8 in 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:11:58.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:11:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:11:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[40056]: osdmap e592: 8 total, 8 up, 8 in 2024-09-16T09:11:58.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:11:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-120"} : dispatch 2024-09-16T09:12:00.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:00 smithi114 ceph-mon[33315]: pgmap v906: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-120"}]': finished 2024-09-16T09:12:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:00 smithi114 ceph-mon[33315]: osdmap e593: 8 total, 8 up, 8 in 2024-09-16T09:12:00.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-120", "mode": "writeback"} : dispatch 2024-09-16T09:12:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:00 smithi111 ceph-mon[29803]: pgmap v906: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-120"}]': finished 2024-09-16T09:12:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:00 smithi111 ceph-mon[29803]: osdmap e593: 8 total, 8 up, 8 in 2024-09-16T09:12:00.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-120", "mode": "writeback"} : dispatch 2024-09-16T09:12:00.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:00 smithi111 ceph-mon[40056]: pgmap v906: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-120"}]': finished 2024-09-16T09:12:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:00 smithi111 ceph-mon[40056]: osdmap e593: 8 total, 8 up, 8 in 2024-09-16T09:12:00.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-120", "mode": "writeback"} : dispatch 2024-09-16T09:12:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:01 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-120", "mode": "writeback"}]': finished 2024-09-16T09:12:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:01 smithi114 ceph-mon[33315]: osdmap e594: 8 total, 8 up, 8 in 2024-09-16T09:12:01.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:01.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:01 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:01.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-120", "mode": "writeback"}]': finished 2024-09-16T09:12:01.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:01 smithi111 ceph-mon[40056]: osdmap e594: 8 total, 8 up, 8 in 2024-09-16T09:12:01.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:01 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-120", "mode": "writeback"}]': finished 2024-09-16T09:12:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:01 smithi111 ceph-mon[29803]: osdmap e594: 8 total, 8 up, 8 in 2024-09-16T09:12:01.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:02.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:12:02 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:12:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:02 smithi114 ceph-mon[33315]: pgmap v909: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:02 smithi114 ceph-mon[33315]: osdmap e595: 8 total, 8 up, 8 in 2024-09-16T09:12:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120"} : dispatch 2024-09-16T09:12:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:02 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:12:02.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:02 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[40056]: pgmap v909: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[40056]: osdmap e595: 8 total, 8 up, 8 in 2024-09-16T09:12:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120"} : dispatch 2024-09-16T09:12:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:12:02.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[29803]: pgmap v909: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[29803]: osdmap e595: 8 total, 8 up, 8 in 2024-09-16T09:12:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120"} : dispatch 2024-09-16T09:12:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:12:02.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:02 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:03 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:03 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120"}]': finished 2024-09-16T09:12:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:03 smithi114 ceph-mon[33315]: osdmap e596: 8 total, 8 up, 8 in 2024-09-16T09:12:03.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:03 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:12:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:12:03.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120"}]': finished 2024-09-16T09:12:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[40056]: osdmap e596: 8 total, 8 up, 8 in 2024-09-16T09:12:03.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-120"}]': finished 2024-09-16T09:12:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[29803]: osdmap e596: 8 total, 8 up, 8 in 2024-09-16T09:12:03.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:03 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:04 smithi114 ceph-mon[33315]: pgmap v912: 204 pgs: 204 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:12:04.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:04 smithi114 ceph-mon[33315]: osdmap e597: 8 total, 8 up, 8 in 2024-09-16T09:12:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:04 smithi111 ceph-mon[29803]: pgmap v912: 204 pgs: 204 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:12:04.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:04 smithi111 ceph-mon[29803]: osdmap e597: 8 total, 8 up, 8 in 2024-09-16T09:12:04.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:04 smithi111 ceph-mon[40056]: pgmap v912: 204 pgs: 204 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-16T09:12:04.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:04 smithi111 ceph-mon[40056]: osdmap e597: 8 total, 8 up, 8 in 2024-09-16T09:12:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:05 smithi114 ceph-mon[33315]: osdmap e598: 8 total, 8 up, 8 in 2024-09-16T09:12:05.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-122","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:05 smithi111 ceph-mon[29803]: osdmap e598: 8 total, 8 up, 8 in 2024-09-16T09:12:05.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-122","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:05.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:05 smithi111 ceph-mon[40056]: osdmap e598: 8 total, 8 up, 8 in 2024-09-16T09:12:05.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-122","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:06 smithi114 ceph-mon[33315]: pgmap v915: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:12:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:06 smithi114 ceph-mon[33315]: osdmap e599: 8 total, 8 up, 8 in 2024-09-16T09:12:06.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:06 smithi111 ceph-mon[29803]: pgmap v915: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:12:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:06 smithi111 ceph-mon[29803]: osdmap e599: 8 total, 8 up, 8 in 2024-09-16T09:12:06.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:06 smithi111 ceph-mon[40056]: pgmap v915: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:12:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:06 smithi111 ceph-mon[40056]: osdmap e599: 8 total, 8 up, 8 in 2024-09-16T09:12:06.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:07 smithi114 ceph-mon[33315]: osdmap e600: 8 total, 8 up, 8 in 2024-09-16T09:12:07.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-122"} : dispatch 2024-09-16T09:12:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:07 smithi111 ceph-mon[29803]: osdmap e600: 8 total, 8 up, 8 in 2024-09-16T09:12:07.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-122"} : dispatch 2024-09-16T09:12:07.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:07.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:07 smithi111 ceph-mon[40056]: osdmap e600: 8 total, 8 up, 8 in 2024-09-16T09:12:07.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-122"} : dispatch 2024-09-16T09:12:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:08 smithi111 ceph-mon[29803]: pgmap v918: 204 pgs: 204 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:12:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-122"}]': finished 2024-09-16T09:12:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:08 smithi111 ceph-mon[29803]: osdmap e601: 8 total, 8 up, 8 in 2024-09-16T09:12:08.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-122", "mode": "writeback"} : dispatch 2024-09-16T09:12:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:08 smithi111 ceph-mon[40056]: pgmap v918: 204 pgs: 204 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:12:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-122"}]': finished 2024-09-16T09:12:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:08 smithi111 ceph-mon[40056]: osdmap e601: 8 total, 8 up, 8 in 2024-09-16T09:12:08.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-122", "mode": "writeback"} : dispatch 2024-09-16T09:12:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:08 smithi114 ceph-mon[33315]: pgmap v918: 204 pgs: 204 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:12:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-122"}]': finished 2024-09-16T09:12:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:08 smithi114 ceph-mon[33315]: osdmap e601: 8 total, 8 up, 8 in 2024-09-16T09:12:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-122", "mode": "writeback"} : dispatch 2024-09-16T09:12:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:09 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:09.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-122", "mode": "writeback"}]': finished 2024-09-16T09:12:09.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:09 smithi114 ceph-mon[33315]: osdmap e602: 8 total, 8 up, 8 in 2024-09-16T09:12:09.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:09 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:09.915 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:09 smithi114 ceph-mon[33315]: pgmap v921: 204 pgs: 204 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:09.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:09.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-122", "mode": "writeback"}]': finished 2024-09-16T09:12:09.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[29803]: osdmap e602: 8 total, 8 up, 8 in 2024-09-16T09:12:09.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:09.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[29803]: pgmap v921: 204 pgs: 204 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-122", "mode": "writeback"}]': finished 2024-09-16T09:12:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[40056]: osdmap e602: 8 total, 8 up, 8 in 2024-09-16T09:12:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:09.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:09 smithi111 ceph-mon[40056]: pgmap v921: 204 pgs: 204 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:10 smithi114 ceph-mon[33315]: osdmap e603: 8 total, 8 up, 8 in 2024-09-16T09:12:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122"} : dispatch 2024-09-16T09:12:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:10 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:10.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122"}]': finished 2024-09-16T09:12:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[29803]: osdmap e603: 8 total, 8 up, 8 in 2024-09-16T09:12:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122"} : dispatch 2024-09-16T09:12:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:10.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122"}]': finished 2024-09-16T09:12:10.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[40056]: osdmap e603: 8 total, 8 up, 8 in 2024-09-16T09:12:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122"} : dispatch 2024-09-16T09:12:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:10.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-122"}]': finished 2024-09-16T09:12:11.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:11 smithi114 ceph-mon[33315]: osdmap e604: 8 total, 8 up, 8 in 2024-09-16T09:12:11.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:11 smithi114 ceph-mon[33315]: pgmap v924: 204 pgs: 204 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:11 smithi111 ceph-mon[29803]: osdmap e604: 8 total, 8 up, 8 in 2024-09-16T09:12:11.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:11 smithi111 ceph-mon[29803]: pgmap v924: 204 pgs: 204 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:11.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:11 smithi111 ceph-mon[40056]: osdmap e604: 8 total, 8 up, 8 in 2024-09-16T09:12:11.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:11 smithi111 ceph-mon[40056]: pgmap v924: 204 pgs: 204 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:12 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:12 smithi114 ceph-mon[33315]: osdmap e605: 8 total, 8 up, 8 in 2024-09-16T09:12:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:12 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:12.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:12:12 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:12:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:12 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:12 smithi111 ceph-mon[29803]: osdmap e605: 8 total, 8 up, 8 in 2024-09-16T09:12:12.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:12 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:12 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:12 smithi111 ceph-mon[40056]: osdmap e605: 8 total, 8 up, 8 in 2024-09-16T09:12:12.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:12 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:13.462 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:12:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:12:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:12:13.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:13 smithi111 ceph-mon[29803]: osdmap e606: 8 total, 8 up, 8 in 2024-09-16T09:12:13.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-124","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:13.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:13 smithi111 ceph-mon[29803]: pgmap v927: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:13.885 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:13 smithi111 ceph-mon[40056]: osdmap e606: 8 total, 8 up, 8 in 2024-09-16T09:12:13.885 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-124","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:13.885 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:13 smithi111 ceph-mon[40056]: pgmap v927: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:13.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:13 smithi114 ceph-mon[33315]: osdmap e606: 8 total, 8 up, 8 in 2024-09-16T09:12:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-124","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:13.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:13 smithi114 ceph-mon[33315]: pgmap v927: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:14.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:14.940 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:14 smithi114 ceph-mon[33315]: osdmap e607: 8 total, 8 up, 8 in 2024-09-16T09:12:14.941 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:14.941 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:14 smithi111 ceph-mon[29803]: osdmap e607: 8 total, 8 up, 8 in 2024-09-16T09:12:14.941 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:14.941 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:14 smithi111 ceph-mon[40056]: osdmap e607: 8 total, 8 up, 8 in 2024-09-16T09:12:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:15 smithi114 ceph-mon[33315]: osdmap e608: 8 total, 8 up, 8 in 2024-09-16T09:12:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:15 smithi114 ceph-mon[33315]: pgmap v930: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:15 smithi111 ceph-mon[40056]: osdmap e608: 8 total, 8 up, 8 in 2024-09-16T09:12:15.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:15.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:15 smithi111 ceph-mon[40056]: pgmap v930: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:15.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:15 smithi111 ceph-mon[29803]: osdmap e608: 8 total, 8 up, 8 in 2024-09-16T09:12:15.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:15.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:15 smithi111 ceph-mon[29803]: pgmap v930: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:16 smithi114 ceph-mon[33315]: osdmap e609: 8 total, 8 up, 8 in 2024-09-16T09:12:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-124"} : dispatch 2024-09-16T09:12:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:12:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:16 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:16.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:16.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[29803]: osdmap e609: 8 total, 8 up, 8 in 2024-09-16T09:12:16.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-124"} : dispatch 2024-09-16T09:12:16.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:12:16.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[40056]: osdmap e609: 8 total, 8 up, 8 in 2024-09-16T09:12:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-124"} : dispatch 2024-09-16T09:12:16.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:12:16.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:16 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-124"}]': finished 2024-09-16T09:12:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:17 smithi114 ceph-mon[33315]: osdmap e610: 8 total, 8 up, 8 in 2024-09-16T09:12:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-124", "mode": "writeback"} : dispatch 2024-09-16T09:12:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:17 smithi114 ceph-mon[33315]: pgmap v933: 204 pgs: 204 active+clean; 583 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-16T09:12:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:17 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-124", "mode": "writeback"}]': finished 2024-09-16T09:12:17.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-124"}]': finished 2024-09-16T09:12:17.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[40056]: osdmap e610: 8 total, 8 up, 8 in 2024-09-16T09:12:17.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-124", "mode": "writeback"} : dispatch 2024-09-16T09:12:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[40056]: pgmap v933: 204 pgs: 204 active+clean; 583 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-16T09:12:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-124", "mode": "writeback"}]': finished 2024-09-16T09:12:17.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-124"}]': finished 2024-09-16T09:12:17.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[29803]: osdmap e610: 8 total, 8 up, 8 in 2024-09-16T09:12:17.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-124", "mode": "writeback"} : dispatch 2024-09-16T09:12:17.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[29803]: pgmap v933: 204 pgs: 204 active+clean; 583 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-16T09:12:17.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:17.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-124", "mode": "writeback"}]': finished 2024-09-16T09:12:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:18 smithi114 ceph-mon[33315]: osdmap e611: 8 total, 8 up, 8 in 2024-09-16T09:12:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:18 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:18.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:18 smithi111 ceph-mon[40056]: osdmap e611: 8 total, 8 up, 8 in 2024-09-16T09:12:18.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:18 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:18.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:18 smithi111 ceph-mon[29803]: osdmap e611: 8 total, 8 up, 8 in 2024-09-16T09:12:18.928 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:18 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:19 smithi114 ceph-mon[33315]: osdmap e612: 8 total, 8 up, 8 in 2024-09-16T09:12:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:19 smithi114 ceph-mon[33315]: pgmap v936: 204 pgs: 204 active+clean; 583 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-16T09:12:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:19 smithi111 ceph-mon[29803]: osdmap e612: 8 total, 8 up, 8 in 2024-09-16T09:12:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:19 smithi111 ceph-mon[29803]: pgmap v936: 204 pgs: 204 active+clean; 583 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-16T09:12:19.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:19 smithi111 ceph-mon[40056]: osdmap e612: 8 total, 8 up, 8 in 2024-09-16T09:12:19.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:19.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:19 smithi111 ceph-mon[40056]: pgmap v936: 204 pgs: 204 active+clean; 583 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-16T09:12:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:20 smithi114 ceph-mon[33315]: osdmap e613: 8 total, 8 up, 8 in 2024-09-16T09:12:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124"} : dispatch 2024-09-16T09:12:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:20 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124"}]': finished 2024-09-16T09:12:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[40056]: osdmap e613: 8 total, 8 up, 8 in 2024-09-16T09:12:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124"} : dispatch 2024-09-16T09:12:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124"}]': finished 2024-09-16T09:12:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[29803]: osdmap e613: 8 total, 8 up, 8 in 2024-09-16T09:12:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124"} : dispatch 2024-09-16T09:12:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:20.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-124"}]': finished 2024-09-16T09:12:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:21 smithi114 ceph-mon[33315]: osdmap e614: 8 total, 8 up, 8 in 2024-09-16T09:12:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:21 smithi114 ceph-mon[33315]: pgmap v939: 204 pgs: 204 active+clean; 583 KiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:21 smithi111 ceph-mon[40056]: osdmap e614: 8 total, 8 up, 8 in 2024-09-16T09:12:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:21 smithi111 ceph-mon[40056]: pgmap v939: 204 pgs: 204 active+clean; 583 KiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:21 smithi111 ceph-mon[29803]: osdmap e614: 8 total, 8 up, 8 in 2024-09-16T09:12:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:21 smithi111 ceph-mon[29803]: pgmap v939: 204 pgs: 204 active+clean; 583 KiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:22.533 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:12:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:12:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:22 smithi114 ceph-mon[33315]: osdmap e615: 8 total, 8 up, 8 in 2024-09-16T09:12:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:22 smithi111 ceph-mon[40056]: osdmap e615: 8 total, 8 up, 8 in 2024-09-16T09:12:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:22 smithi111 ceph-mon[29803]: osdmap e615: 8 total, 8 up, 8 in 2024-09-16T09:12:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:23.544 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:12:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:12:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:12:23.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:23 smithi114 ceph-mon[33315]: osdmap e616: 8 total, 8 up, 8 in 2024-09-16T09:12:23.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-126","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:23.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:23 smithi114 ceph-mon[33315]: pgmap v942: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:23.816 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:23 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:23 smithi111 ceph-mon[29803]: osdmap e616: 8 total, 8 up, 8 in 2024-09-16T09:12:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-126","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:23 smithi111 ceph-mon[29803]: pgmap v942: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:23 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:23 smithi111 ceph-mon[40056]: osdmap e616: 8 total, 8 up, 8 in 2024-09-16T09:12:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-126","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:23 smithi111 ceph-mon[40056]: pgmap v942: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:23 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:24 smithi114 ceph-mon[33315]: osdmap e617: 8 total, 8 up, 8 in 2024-09-16T09:12:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:24.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:24 smithi111 ceph-mon[29803]: osdmap e617: 8 total, 8 up, 8 in 2024-09-16T09:12:24.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:24.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:24.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:24 smithi111 ceph-mon[40056]: osdmap e617: 8 total, 8 up, 8 in 2024-09-16T09:12:24.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:25 smithi114 ceph-mon[33315]: osdmap e618: 8 total, 8 up, 8 in 2024-09-16T09:12:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-126"} : dispatch 2024-09-16T09:12:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:25 smithi114 ceph-mon[33315]: pgmap v945: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-126"}]': finished 2024-09-16T09:12:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:25 smithi114 ceph-mon[33315]: osdmap e619: 8 total, 8 up, 8 in 2024-09-16T09:12:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-126", "mode": "writeback"} : dispatch 2024-09-16T09:12:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[29803]: osdmap e618: 8 total, 8 up, 8 in 2024-09-16T09:12:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-126"} : dispatch 2024-09-16T09:12:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[29803]: pgmap v945: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-126"}]': finished 2024-09-16T09:12:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[29803]: osdmap e619: 8 total, 8 up, 8 in 2024-09-16T09:12:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-126", "mode": "writeback"} : dispatch 2024-09-16T09:12:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[40056]: osdmap e618: 8 total, 8 up, 8 in 2024-09-16T09:12:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-126"} : dispatch 2024-09-16T09:12:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[40056]: pgmap v945: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-126"}]': finished 2024-09-16T09:12:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[40056]: osdmap e619: 8 total, 8 up, 8 in 2024-09-16T09:12:25.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-126", "mode": "writeback"} : dispatch 2024-09-16T09:12:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:26 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-126", "mode": "writeback"}]': finished 2024-09-16T09:12:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:26 smithi114 ceph-mon[33315]: osdmap e620: 8 total, 8 up, 8 in 2024-09-16T09:12:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:26 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-126", "mode": "writeback"}]': finished 2024-09-16T09:12:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:26 smithi111 ceph-mon[29803]: osdmap e620: 8 total, 8 up, 8 in 2024-09-16T09:12:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:26 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-126", "mode": "writeback"}]': finished 2024-09-16T09:12:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:26 smithi111 ceph-mon[40056]: osdmap e620: 8 total, 8 up, 8 in 2024-09-16T09:12:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:27 smithi114 ceph-mon[33315]: pgmap v948: 204 pgs: 204 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:27 smithi111 ceph-mon[29803]: pgmap v948: 204 pgs: 204 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:27.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:27 smithi111 ceph-mon[40056]: pgmap v948: 204 pgs: 204 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:28 smithi114 ceph-mon[33315]: osdmap e621: 8 total, 8 up, 8 in 2024-09-16T09:12:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126"} : dispatch 2024-09-16T09:12:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:28 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:28 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126"}]': finished 2024-09-16T09:12:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:28 smithi114 ceph-mon[33315]: osdmap e622: 8 total, 8 up, 8 in 2024-09-16T09:12:28.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:28.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[29803]: osdmap e621: 8 total, 8 up, 8 in 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126"} : dispatch 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126"}]': finished 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[29803]: osdmap e622: 8 total, 8 up, 8 in 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[40056]: osdmap e621: 8 total, 8 up, 8 in 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126"} : dispatch 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:28.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:28.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-126"}]': finished 2024-09-16T09:12:28.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:28 smithi111 ceph-mon[40056]: osdmap e622: 8 total, 8 up, 8 in 2024-09-16T09:12:29.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:12:29.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9073 ms) 2024-09-16T09:12:29.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-16T09:12:29.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:12:29.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6030 ms) 2024-09-16T09:12:29.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-16T09:12:29.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:12:29.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5040 ms) 2024-09-16T09:12:29.362 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26081 ms) 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13373 ms) 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12056 ms) 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: 00000012 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12067 ms) 2024-09-16T09:12:29.363 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6041 ms) 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (555681 ms total) 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1013 ms) 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11049 ms) 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18073 ms) 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-16T09:12:29.364 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30137 ms total) 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7054 ms) 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8056 ms) 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: waiting for scrub... 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: done waiting 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24197 ms) 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9999 ms) 2024-09-16T09:12:29.365 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-16T09:12:29.366 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7990 ms) 2024-09-16T09:12:29.366 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-16T09:12:29.366 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8059 ms) 2024-09-16T09:12:29.366 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-16T09:12:29.366 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10097 ms) 2024-09-16T09:12:29.366 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-16T09:12:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:30 smithi114 ceph-mon[33315]: pgmap v951: 204 pgs: 204 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:30.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:30 smithi114 ceph-mon[33315]: osdmap e623: 8 total, 8 up, 8 in 2024-09-16T09:12:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:30 smithi111 ceph-mon[29803]: pgmap v951: 204 pgs: 204 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:30.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:30 smithi111 ceph-mon[29803]: osdmap e623: 8 total, 8 up, 8 in 2024-09-16T09:12:30.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:30 smithi111 ceph-mon[40056]: pgmap v951: 204 pgs: 204 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:30.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:30 smithi111 ceph-mon[40056]: osdmap e623: 8 total, 8 up, 8 in 2024-09-16T09:12:31.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:31 smithi114 ceph-mon[33315]: osdmap e624: 8 total, 8 up, 8 in 2024-09-16T09:12:31.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-128","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:31 smithi111 ceph-mon[29803]: osdmap e624: 8 total, 8 up, 8 in 2024-09-16T09:12:31.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-128","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:31 smithi111 ceph-mon[40056]: osdmap e624: 8 total, 8 up, 8 in 2024-09-16T09:12:31.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-128","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:32.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:12:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:12:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:32 smithi114 ceph-mon[33315]: pgmap v954: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:32 smithi114 ceph-mon[33315]: osdmap e625: 8 total, 8 up, 8 in 2024-09-16T09:12:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:32.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[29803]: pgmap v954: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[29803]: osdmap e625: 8 total, 8 up, 8 in 2024-09-16T09:12:32.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:32.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[40056]: pgmap v954: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[40056]: osdmap e625: 8 total, 8 up, 8 in 2024-09-16T09:12:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:32.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:33 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:33 smithi114 ceph-mon[33315]: osdmap e626: 8 total, 8 up, 8 in 2024-09-16T09:12:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-128"} : dispatch 2024-09-16T09:12:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:33 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-128"}]': finished 2024-09-16T09:12:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:33 smithi114 ceph-mon[33315]: osdmap e627: 8 total, 8 up, 8 in 2024-09-16T09:12:33.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-128", "mode": "writeback"} : dispatch 2024-09-16T09:12:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:12:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:12:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:33.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[29803]: osdmap e626: 8 total, 8 up, 8 in 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-128"} : dispatch 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-128"}]': finished 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[29803]: osdmap e627: 8 total, 8 up, 8 in 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-128", "mode": "writeback"} : dispatch 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[40056]: osdmap e626: 8 total, 8 up, 8 in 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-128"} : dispatch 2024-09-16T09:12:33.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-128"}]': finished 2024-09-16T09:12:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[40056]: osdmap e627: 8 total, 8 up, 8 in 2024-09-16T09:12:33.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-128", "mode": "writeback"} : dispatch 2024-09-16T09:12:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:34 smithi114 ceph-mon[33315]: pgmap v957: 204 pgs: 9 creating+activating, 195 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:34 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-128", "mode": "writeback"}]': finished 2024-09-16T09:12:34.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:34 smithi114 ceph-mon[33315]: osdmap e628: 8 total, 8 up, 8 in 2024-09-16T09:12:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:34 smithi111 ceph-mon[29803]: pgmap v957: 204 pgs: 9 creating+activating, 195 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:34 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-128", "mode": "writeback"}]': finished 2024-09-16T09:12:34.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:34 smithi111 ceph-mon[29803]: osdmap e628: 8 total, 8 up, 8 in 2024-09-16T09:12:34.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:34 smithi111 ceph-mon[40056]: pgmap v957: 204 pgs: 9 creating+activating, 195 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:34 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-128", "mode": "writeback"}]': finished 2024-09-16T09:12:34.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:34 smithi111 ceph-mon[40056]: osdmap e628: 8 total, 8 up, 8 in 2024-09-16T09:12:36.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:36 smithi114 ceph-mon[33315]: pgmap v960: 204 pgs: 9 creating+activating, 195 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:36.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:36 smithi111 ceph-mon[29803]: pgmap v960: 204 pgs: 9 creating+activating, 195 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:36.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:36 smithi111 ceph-mon[40056]: pgmap v960: 204 pgs: 9 creating+activating, 195 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:38 smithi114 ceph-mon[33315]: pgmap v961: 204 pgs: 204 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 576 B/s wr, 2 op/s 2024-09-16T09:12:38.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:38 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:38 smithi111 ceph-mon[29803]: pgmap v961: 204 pgs: 204 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 576 B/s wr, 2 op/s 2024-09-16T09:12:38.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:38 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:38.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:38 smithi111 ceph-mon[40056]: pgmap v961: 204 pgs: 204 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 576 B/s wr, 2 op/s 2024-09-16T09:12:38.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:38 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:12:40.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:40 smithi114 ceph-mon[33315]: pgmap v962: 204 pgs: 204 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 323 B/s wr, 1 op/s 2024-09-16T09:12:40.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:40 smithi111 ceph-mon[29803]: pgmap v962: 204 pgs: 204 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 323 B/s wr, 1 op/s 2024-09-16T09:12:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:40.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:40 smithi111 ceph-mon[40056]: pgmap v962: 204 pgs: 204 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 323 B/s wr, 1 op/s 2024-09-16T09:12:40.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:41 smithi111 ceph-mon[29803]: osdmap e629: 8 total, 8 up, 8 in 2024-09-16T09:12:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128"} : dispatch 2024-09-16T09:12:41.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:41 smithi111 ceph-mon[40056]: osdmap e629: 8 total, 8 up, 8 in 2024-09-16T09:12:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128"} : dispatch 2024-09-16T09:12:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:41 smithi114 ceph-mon[33315]: osdmap e629: 8 total, 8 up, 8 in 2024-09-16T09:12:41.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128"} : dispatch 2024-09-16T09:12:42.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:12:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:12:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:42 smithi114 ceph-mon[33315]: pgmap v964: 204 pgs: 204 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 839 B/s rd, 279 B/s wr, 1 op/s 2024-09-16T09:12:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:42 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:42 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128"}]': finished 2024-09-16T09:12:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:42 smithi114 ceph-mon[33315]: osdmap e630: 8 total, 8 up, 8 in 2024-09-16T09:12:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[29803]: pgmap v964: 204 pgs: 204 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 839 B/s rd, 279 B/s wr, 1 op/s 2024-09-16T09:12:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128"}]': finished 2024-09-16T09:12:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[29803]: osdmap e630: 8 total, 8 up, 8 in 2024-09-16T09:12:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[40056]: pgmap v964: 204 pgs: 204 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 839 B/s rd, 279 B/s wr, 1 op/s 2024-09-16T09:12:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-128"}]': finished 2024-09-16T09:12:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[40056]: osdmap e630: 8 total, 8 up, 8 in 2024-09-16T09:12:42.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:43.437 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:12:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:12:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:12:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:43 smithi114 ceph-mon[33315]: osdmap e631: 8 total, 8 up, 8 in 2024-09-16T09:12:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:43 smithi114 ceph-mon[33315]: pgmap v967: 172 pgs: 172 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T09:12:43.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:43 smithi111 ceph-mon[29803]: osdmap e631: 8 total, 8 up, 8 in 2024-09-16T09:12:43.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:43 smithi111 ceph-mon[29803]: pgmap v967: 172 pgs: 172 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T09:12:43.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:43 smithi111 ceph-mon[40056]: osdmap e631: 8 total, 8 up, 8 in 2024-09-16T09:12:43.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:43 smithi111 ceph-mon[40056]: pgmap v967: 172 pgs: 172 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-16T09:12:44.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:44 smithi114 ceph-mon[33315]: osdmap e632: 8 total, 8 up, 8 in 2024-09-16T09:12:44.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-130","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:44.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:44 smithi111 ceph-mon[29803]: osdmap e632: 8 total, 8 up, 8 in 2024-09-16T09:12:44.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-130","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:44 smithi111 ceph-mon[40056]: osdmap e632: 8 total, 8 up, 8 in 2024-09-16T09:12:44.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-130","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:45 smithi114 ceph-mon[33315]: osdmap e633: 8 total, 8 up, 8 in 2024-09-16T09:12:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:45.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:45 smithi114 ceph-mon[33315]: pgmap v970: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:12:45.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:45.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:45 smithi111 ceph-mon[29803]: osdmap e633: 8 total, 8 up, 8 in 2024-09-16T09:12:45.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:45.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:45 smithi111 ceph-mon[29803]: pgmap v970: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:12:45.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:45.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:45 smithi111 ceph-mon[40056]: osdmap e633: 8 total, 8 up, 8 in 2024-09-16T09:12:45.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:45.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:45 smithi111 ceph-mon[40056]: pgmap v970: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:12:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:46 smithi114 ceph-mon[33315]: osdmap e634: 8 total, 8 up, 8 in 2024-09-16T09:12:46.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-130"} : dispatch 2024-09-16T09:12:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:46 smithi111 ceph-mon[29803]: osdmap e634: 8 total, 8 up, 8 in 2024-09-16T09:12:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-130"} : dispatch 2024-09-16T09:12:46.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:46.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:46 smithi111 ceph-mon[40056]: osdmap e634: 8 total, 8 up, 8 in 2024-09-16T09:12:46.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-130"} : dispatch 2024-09-16T09:12:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-130"}]': finished 2024-09-16T09:12:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:47 smithi114 ceph-mon[33315]: osdmap e635: 8 total, 8 up, 8 in 2024-09-16T09:12:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-130", "mode": "writeback"} : dispatch 2024-09-16T09:12:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:47 smithi114 ceph-mon[33315]: pgmap v973: 204 pgs: 204 active+clean; 583 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:47 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-130", "mode": "writeback"}]': finished 2024-09-16T09:12:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:47 smithi114 ceph-mon[33315]: osdmap e636: 8 total, 8 up, 8 in 2024-09-16T09:12:47.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-130"}]': finished 2024-09-16T09:12:47.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[29803]: osdmap e635: 8 total, 8 up, 8 in 2024-09-16T09:12:47.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-130", "mode": "writeback"} : dispatch 2024-09-16T09:12:47.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:47.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[29803]: pgmap v973: 204 pgs: 204 active+clean; 583 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:47.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:47.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-130", "mode": "writeback"}]': finished 2024-09-16T09:12:47.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[29803]: osdmap e636: 8 total, 8 up, 8 in 2024-09-16T09:12:47.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-130"}]': finished 2024-09-16T09:12:47.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[40056]: osdmap e635: 8 total, 8 up, 8 in 2024-09-16T09:12:47.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-130", "mode": "writeback"} : dispatch 2024-09-16T09:12:47.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:12:47.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[40056]: pgmap v973: 204 pgs: 204 active+clean; 583 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:47.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:47.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-130", "mode": "writeback"}]': finished 2024-09-16T09:12:47.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:47 smithi111 ceph-mon[40056]: osdmap e636: 8 total, 8 up, 8 in 2024-09-16T09:12:48.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:48.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:49 smithi114 ceph-mon[33315]: osdmap e637: 8 total, 8 up, 8 in 2024-09-16T09:12:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130"} : dispatch 2024-09-16T09:12:49.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:49 smithi114 ceph-mon[33315]: pgmap v976: 204 pgs: 204 active+clean; 583 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:49.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:49 smithi111 ceph-mon[29803]: osdmap e637: 8 total, 8 up, 8 in 2024-09-16T09:12:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130"} : dispatch 2024-09-16T09:12:49.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:49 smithi111 ceph-mon[29803]: pgmap v976: 204 pgs: 204 active+clean; 583 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:49 smithi111 ceph-mon[40056]: osdmap e637: 8 total, 8 up, 8 in 2024-09-16T09:12:49.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130"} : dispatch 2024-09-16T09:12:49.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:49 smithi111 ceph-mon[40056]: pgmap v976: 204 pgs: 204 active+clean; 583 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:12:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:50 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130"}]': finished 2024-09-16T09:12:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:50 smithi114 ceph-mon[33315]: osdmap e638: 8 total, 8 up, 8 in 2024-09-16T09:12:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:50 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130"}]': finished 2024-09-16T09:12:50.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:50 smithi111 ceph-mon[29803]: osdmap e638: 8 total, 8 up, 8 in 2024-09-16T09:12:50.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:50 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:12:50.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-130"}]': finished 2024-09-16T09:12:50.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:50 smithi111 ceph-mon[40056]: osdmap e638: 8 total, 8 up, 8 in 2024-09-16T09:12:51.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:51 smithi114 ceph-mon[33315]: osdmap e639: 8 total, 8 up, 8 in 2024-09-16T09:12:51.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:51 smithi114 ceph-mon[33315]: pgmap v979: 172 pgs: 172 active+clean; 583 KiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:51.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:51 smithi114 ceph-mon[33315]: osdmap e640: 8 total, 8 up, 8 in 2024-09-16T09:12:51.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-132","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:51 smithi111 ceph-mon[29803]: osdmap e639: 8 total, 8 up, 8 in 2024-09-16T09:12:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:51 smithi111 ceph-mon[29803]: pgmap v979: 172 pgs: 172 active+clean; 583 KiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:51 smithi111 ceph-mon[29803]: osdmap e640: 8 total, 8 up, 8 in 2024-09-16T09:12:51.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-132","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:51 smithi111 ceph-mon[40056]: osdmap e639: 8 total, 8 up, 8 in 2024-09-16T09:12:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:51 smithi111 ceph-mon[40056]: pgmap v979: 172 pgs: 172 active+clean; 583 KiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:12:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:51 smithi111 ceph-mon[40056]: osdmap e640: 8 total, 8 up, 8 in 2024-09-16T09:12:51.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-132","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:12:52.528 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:12:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:12:52.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:52 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:52.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:52.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:52 smithi114 ceph-mon[33315]: osdmap e641: 8 total, 8 up, 8 in 2024-09-16T09:12:52.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:52 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:52 smithi111 ceph-mon[29803]: osdmap e641: 8 total, 8 up, 8 in 2024-09-16T09:12:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:52.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:52 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:12:52.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:12:52.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:52 smithi111 ceph-mon[40056]: osdmap e641: 8 total, 8 up, 8 in 2024-09-16T09:12:52.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:12:53.528 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:12:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:12:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:12:53.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:53 smithi114 ceph-mon[33315]: pgmap v982: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:53.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:53.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:53 smithi114 ceph-mon[33315]: osdmap e642: 8 total, 8 up, 8 in 2024-09-16T09:12:53.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"} : dispatch 2024-09-16T09:12:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:53 smithi111 ceph-mon[29803]: pgmap v982: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:53.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:53.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:53 smithi111 ceph-mon[29803]: osdmap e642: 8 total, 8 up, 8 in 2024-09-16T09:12:53.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"} : dispatch 2024-09-16T09:12:53.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:53 smithi111 ceph-mon[40056]: pgmap v982: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:53.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:12:53.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:53 smithi111 ceph-mon[40056]: osdmap e642: 8 total, 8 up, 8 in 2024-09-16T09:12:53.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"} : dispatch 2024-09-16T09:12:54.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:54 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:12:54.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"}]': finished 2024-09-16T09:12:54.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:54 smithi114 ceph-mon[33315]: osdmap e643: 8 total, 8 up, 8 in 2024-09-16T09:12:54.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-132", "mode": "writeback"} : dispatch 2024-09-16T09:12:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:54 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:12:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"}]': finished 2024-09-16T09:12:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:54 smithi111 ceph-mon[29803]: osdmap e643: 8 total, 8 up, 8 in 2024-09-16T09:12:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-132", "mode": "writeback"} : dispatch 2024-09-16T09:12:54.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:54 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:12:54.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"}]': finished 2024-09-16T09:12:54.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:54 smithi111 ceph-mon[40056]: osdmap e643: 8 total, 8 up, 8 in 2024-09-16T09:12:54.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-132", "mode": "writeback"} : dispatch 2024-09-16T09:12:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:55 smithi114 ceph-mon[33315]: pgmap v985: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:55 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:12:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:55 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:12:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:55 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:12:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:55 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-132", "mode": "writeback"}]': finished 2024-09-16T09:12:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:55 smithi114 ceph-mon[33315]: osdmap e644: 8 total, 8 up, 8 in 2024-09-16T09:12:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[29803]: pgmap v985: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:12:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:12:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:12:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:55.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-132", "mode": "writeback"}]': finished 2024-09-16T09:12:55.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[29803]: osdmap e644: 8 total, 8 up, 8 in 2024-09-16T09:12:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[40056]: pgmap v985: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:12:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:12:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:12:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:12:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:12:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-132", "mode": "writeback"}]': finished 2024-09-16T09:12:55.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:55 smithi111 ceph-mon[40056]: osdmap e644: 8 total, 8 up, 8 in 2024-09-16T09:12:57.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:57 smithi114 ceph-mon[33315]: osdmap e645: 8 total, 8 up, 8 in 2024-09-16T09:12:57.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:57 smithi114 ceph-mon[33315]: pgmap v988: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:12:57.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:57 smithi111 ceph-mon[40056]: osdmap e645: 8 total, 8 up, 8 in 2024-09-16T09:12:57.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:57 smithi111 ceph-mon[40056]: pgmap v988: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:12:57.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:57 smithi111 ceph-mon[29803]: osdmap e645: 8 total, 8 up, 8 in 2024-09-16T09:12:57.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:57 smithi111 ceph-mon[29803]: pgmap v988: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:12:58.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:58 smithi114 ceph-mon[33315]: osdmap e646: 8 total, 8 up, 8 in 2024-09-16T09:12:58.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:58.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:58 smithi111 ceph-mon[40056]: osdmap e646: 8 total, 8 up, 8 in 2024-09-16T09:12:58.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:58.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:58 smithi111 ceph-mon[29803]: osdmap e646: 8 total, 8 up, 8 in 2024-09-16T09:12:58.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:12:59.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:59.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:59 smithi114 ceph-mon[33315]: osdmap e647: 8 total, 8 up, 8 in 2024-09-16T09:12:59.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"} : dispatch 2024-09-16T09:12:59.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:12:59 smithi114 ceph-mon[33315]: pgmap v991: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:12:59.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:59.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:59 smithi111 ceph-mon[29803]: osdmap e647: 8 total, 8 up, 8 in 2024-09-16T09:12:59.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"} : dispatch 2024-09-16T09:12:59.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:12:59 smithi111 ceph-mon[29803]: pgmap v991: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:12:59.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:12:59.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:59 smithi111 ceph-mon[40056]: osdmap e647: 8 total, 8 up, 8 in 2024-09-16T09:12:59.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"} : dispatch 2024-09-16T09:12:59.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:12:59 smithi111 ceph-mon[40056]: pgmap v991: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:13:00.919 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"}]': finished 2024-09-16T09:13:00.919 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:00 smithi114 ceph-mon[33315]: osdmap e648: 8 total, 8 up, 8 in 2024-09-16T09:13:00.920 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:00 smithi114 ceph-mon[33315]: osdmap e649: 8 total, 8 up, 8 in 2024-09-16T09:13:00.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"}]': finished 2024-09-16T09:13:00.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:00 smithi111 ceph-mon[40056]: osdmap e648: 8 total, 8 up, 8 in 2024-09-16T09:13:00.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:00 smithi111 ceph-mon[40056]: osdmap e649: 8 total, 8 up, 8 in 2024-09-16T09:13:00.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-132"}]': finished 2024-09-16T09:13:00.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:00 smithi111 ceph-mon[29803]: osdmap e648: 8 total, 8 up, 8 in 2024-09-16T09:13:00.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:00 smithi111 ceph-mon[29803]: osdmap e649: 8 total, 8 up, 8 in 2024-09-16T09:13:01.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:01.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:01 smithi114 ceph-mon[33315]: pgmap v994: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:01.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:13:01.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:01 smithi111 ceph-mon[29803]: pgmap v994: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:13:01.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:01.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:01.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:01 smithi111 ceph-mon[40056]: pgmap v994: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:01.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:13:01.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:02.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:13:02 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:13:02.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:13:02.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:02 smithi111 ceph-mon[29803]: osdmap e650: 8 total, 8 up, 8 in 2024-09-16T09:13:02.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132"} : dispatch 2024-09-16T09:13:02.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:02.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:13:02.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:02 smithi111 ceph-mon[40056]: osdmap e650: 8 total, 8 up, 8 in 2024-09-16T09:13:02.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132"} : dispatch 2024-09-16T09:13:02.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:13:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:02 smithi114 ceph-mon[33315]: osdmap e650: 8 total, 8 up, 8 in 2024-09-16T09:13:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132"} : dispatch 2024-09-16T09:13:02.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:03.556 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:13:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:13:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:13:03.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:03 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:03.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132"}]': finished 2024-09-16T09:13:03.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:03 smithi114 ceph-mon[33315]: osdmap e651: 8 total, 8 up, 8 in 2024-09-16T09:13:03.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:03 smithi114 ceph-mon[33315]: pgmap v997: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-16T09:13:03.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:03 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:03.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132"}]': finished 2024-09-16T09:13:03.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:03 smithi111 ceph-mon[29803]: osdmap e651: 8 total, 8 up, 8 in 2024-09-16T09:13:03.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:03 smithi111 ceph-mon[29803]: pgmap v997: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-16T09:13:03.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:03 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:03.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-132"}]': finished 2024-09-16T09:13:03.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:03 smithi111 ceph-mon[40056]: osdmap e651: 8 total, 8 up, 8 in 2024-09-16T09:13:03.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:03 smithi111 ceph-mon[40056]: pgmap v997: 204 pgs: 204 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-16T09:13:04.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:04 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:04.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:04 smithi114 ceph-mon[33315]: osdmap e652: 8 total, 8 up, 8 in 2024-09-16T09:13:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:04 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:04.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:04 smithi111 ceph-mon[29803]: osdmap e652: 8 total, 8 up, 8 in 2024-09-16T09:13:04.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:04 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:04.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:04 smithi111 ceph-mon[40056]: osdmap e652: 8 total, 8 up, 8 in 2024-09-16T09:13:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:05 smithi114 ceph-mon[33315]: osdmap e653: 8 total, 8 up, 8 in 2024-09-16T09:13:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-134","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:05.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:05 smithi114 ceph-mon[33315]: pgmap v1000: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:13:05.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:05 smithi111 ceph-mon[29803]: osdmap e653: 8 total, 8 up, 8 in 2024-09-16T09:13:05.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-134","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:05.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:05 smithi111 ceph-mon[29803]: pgmap v1000: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:13:05.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:05 smithi111 ceph-mon[40056]: osdmap e653: 8 total, 8 up, 8 in 2024-09-16T09:13:05.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-134","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:05.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:05 smithi111 ceph-mon[40056]: pgmap v1000: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 674 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:13:06.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:06 smithi114 ceph-mon[33315]: osdmap e654: 8 total, 8 up, 8 in 2024-09-16T09:13:06.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:06 smithi111 ceph-mon[29803]: osdmap e654: 8 total, 8 up, 8 in 2024-09-16T09:13:06.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:06.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:06.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:06 smithi111 ceph-mon[40056]: osdmap e654: 8 total, 8 up, 8 in 2024-09-16T09:13:06.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:07 smithi114 ceph-mon[33315]: osdmap e655: 8 total, 8 up, 8 in 2024-09-16T09:13:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:07 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-134"} : dispatch 2024-09-16T09:13:07.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:07 smithi114 ceph-mon[33315]: pgmap v1003: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:13:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:07 smithi111 ceph-mon[29803]: osdmap e655: 8 total, 8 up, 8 in 2024-09-16T09:13:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:07 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-134"} : dispatch 2024-09-16T09:13:07.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:07 smithi111 ceph-mon[29803]: pgmap v1003: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:13:07.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:07.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:07 smithi111 ceph-mon[40056]: osdmap e655: 8 total, 8 up, 8 in 2024-09-16T09:13:07.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:07 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-134"} : dispatch 2024-09-16T09:13:07.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:07 smithi111 ceph-mon[40056]: pgmap v1003: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-16T09:13:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-134"}]': finished 2024-09-16T09:13:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:08 smithi114 ceph-mon[33315]: osdmap e656: 8 total, 8 up, 8 in 2024-09-16T09:13:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-134", "mode": "writeback"} : dispatch 2024-09-16T09:13:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:08 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:13:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-134", "mode": "writeback"}]': finished 2024-09-16T09:13:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:08 smithi114 ceph-mon[33315]: osdmap e657: 8 total, 8 up, 8 in 2024-09-16T09:13:08.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:08 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-134"}]': finished 2024-09-16T09:13:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[29803]: osdmap e656: 8 total, 8 up, 8 in 2024-09-16T09:13:08.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-134", "mode": "writeback"} : dispatch 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-134", "mode": "writeback"}]': finished 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[29803]: osdmap e657: 8 total, 8 up, 8 in 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-134"}]': finished 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[40056]: osdmap e656: 8 total, 8 up, 8 in 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-134", "mode": "writeback"} : dispatch 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-134", "mode": "writeback"}]': finished 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[40056]: osdmap e657: 8 total, 8 up, 8 in 2024-09-16T09:13:08.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:08 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:10 smithi114 ceph-mon[33315]: pgmap v1006: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:13:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:10 smithi114 ceph-mon[33315]: osdmap e658: 8 total, 8 up, 8 in 2024-09-16T09:13:10.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134"} : dispatch 2024-09-16T09:13:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:10 smithi111 ceph-mon[29803]: pgmap v1006: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:13:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:10 smithi111 ceph-mon[29803]: osdmap e658: 8 total, 8 up, 8 in 2024-09-16T09:13:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134"} : dispatch 2024-09-16T09:13:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:10 smithi111 ceph-mon[40056]: pgmap v1006: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:13:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:10 smithi111 ceph-mon[40056]: osdmap e658: 8 total, 8 up, 8 in 2024-09-16T09:13:10.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134"} : dispatch 2024-09-16T09:13:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:11 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134"}]': finished 2024-09-16T09:13:11.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:11 smithi114 ceph-mon[33315]: osdmap e659: 8 total, 8 up, 8 in 2024-09-16T09:13:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:11 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134"}]': finished 2024-09-16T09:13:11.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:11 smithi111 ceph-mon[29803]: osdmap e659: 8 total, 8 up, 8 in 2024-09-16T09:13:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:11 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-134"}]': finished 2024-09-16T09:13:11.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:11 smithi111 ceph-mon[40056]: osdmap e659: 8 total, 8 up, 8 in 2024-09-16T09:13:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:12 smithi114 ceph-mon[33315]: pgmap v1009: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:12 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:12 smithi114 ceph-mon[33315]: osdmap e660: 8 total, 8 up, 8 in 2024-09-16T09:13:12.665 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:13:12 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:13:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:12 smithi111 ceph-mon[29803]: pgmap v1009: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:12 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:12.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:12 smithi111 ceph-mon[29803]: osdmap e660: 8 total, 8 up, 8 in 2024-09-16T09:13:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:12 smithi111 ceph-mon[40056]: pgmap v1009: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:12 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:12.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:12 smithi111 ceph-mon[40056]: osdmap e660: 8 total, 8 up, 8 in 2024-09-16T09:13:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:13 smithi114 ceph-mon[33315]: osdmap e661: 8 total, 8 up, 8 in 2024-09-16T09:13:13.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-136","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:13.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:13.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:13 smithi111 ceph-mon[40056]: osdmap e661: 8 total, 8 up, 8 in 2024-09-16T09:13:13.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-136","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:13 smithi111 ceph-mon[29803]: osdmap e661: 8 total, 8 up, 8 in 2024-09-16T09:13:13.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-136","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:13.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:13:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:13:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:13:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:14 smithi114 ceph-mon[33315]: pgmap v1012: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:13:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:14 smithi114 ceph-mon[33315]: osdmap e662: 8 total, 8 up, 8 in 2024-09-16T09:13:14.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:14 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:14 smithi111 ceph-mon[40056]: pgmap v1012: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:13:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:14 smithi111 ceph-mon[40056]: osdmap e662: 8 total, 8 up, 8 in 2024-09-16T09:13:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:14 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:14 smithi111 ceph-mon[29803]: pgmap v1012: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:13:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:14 smithi111 ceph-mon[29803]: osdmap e662: 8 total, 8 up, 8 in 2024-09-16T09:13:14.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:14 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:15.681 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:15.681 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:15 smithi111 ceph-mon[40056]: osdmap e663: 8 total, 8 up, 8 in 2024-09-16T09:13:15.681 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-136"} : dispatch 2024-09-16T09:13:15.681 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-136"}]': finished 2024-09-16T09:13:15.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:15.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:15 smithi111 ceph-mon[29803]: osdmap e663: 8 total, 8 up, 8 in 2024-09-16T09:13:15.681 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-136"} : dispatch 2024-09-16T09:13:15.682 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-136"}]': finished 2024-09-16T09:13:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:15 smithi114 ceph-mon[33315]: osdmap e663: 8 total, 8 up, 8 in 2024-09-16T09:13:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-136"} : dispatch 2024-09-16T09:13:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-136"}]': finished 2024-09-16T09:13:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[40056]: pgmap v1015: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:13:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[40056]: osdmap e664: 8 total, 8 up, 8 in 2024-09-16T09:13:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-136", "mode": "writeback"} : dispatch 2024-09-16T09:13:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:13:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-136", "mode": "writeback"}]': finished 2024-09-16T09:13:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[40056]: osdmap e665: 8 total, 8 up, 8 in 2024-09-16T09:13:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[29803]: pgmap v1015: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:13:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[29803]: osdmap e664: 8 total, 8 up, 8 in 2024-09-16T09:13:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-136", "mode": "writeback"} : dispatch 2024-09-16T09:13:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:13:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-136", "mode": "writeback"}]': finished 2024-09-16T09:13:16.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:16 smithi111 ceph-mon[29803]: osdmap e665: 8 total, 8 up, 8 in 2024-09-16T09:13:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:16 smithi114 ceph-mon[33315]: pgmap v1015: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-16T09:13:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:16 smithi114 ceph-mon[33315]: osdmap e664: 8 total, 8 up, 8 in 2024-09-16T09:13:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-136", "mode": "writeback"} : dispatch 2024-09-16T09:13:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:16 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:13:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-136", "mode": "writeback"}]': finished 2024-09-16T09:13:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:16 smithi114 ceph-mon[33315]: osdmap e665: 8 total, 8 up, 8 in 2024-09-16T09:13:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:17.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:17.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[29803]: pgmap v1018: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-09-16T09:13:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:13:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[29803]: osdmap e666: 8 total, 8 up, 8 in 2024-09-16T09:13:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136"} : dispatch 2024-09-16T09:13:18.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[40056]: pgmap v1018: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-09-16T09:13:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:13:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[40056]: osdmap e666: 8 total, 8 up, 8 in 2024-09-16T09:13:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136"} : dispatch 2024-09-16T09:13:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:18 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:18 smithi114 ceph-mon[33315]: pgmap v1018: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-09-16T09:13:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:13:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:18 smithi114 ceph-mon[33315]: osdmap e666: 8 total, 8 up, 8 in 2024-09-16T09:13:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136"} : dispatch 2024-09-16T09:13:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:18 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:19.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:19 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136"}]': finished 2024-09-16T09:13:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:19 smithi111 ceph-mon[29803]: osdmap e667: 8 total, 8 up, 8 in 2024-09-16T09:13:19.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:19 smithi111 ceph-mon[29803]: pgmap v1021: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-09-16T09:13:19.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:19 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136"}]': finished 2024-09-16T09:13:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:19 smithi111 ceph-mon[40056]: osdmap e667: 8 total, 8 up, 8 in 2024-09-16T09:13:19.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:19 smithi111 ceph-mon[40056]: pgmap v1021: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-09-16T09:13:19.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:19 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-136"}]': finished 2024-09-16T09:13:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:19 smithi114 ceph-mon[33315]: osdmap e667: 8 total, 8 up, 8 in 2024-09-16T09:13:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:19 smithi114 ceph-mon[33315]: pgmap v1021: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-09-16T09:13:20.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:20 smithi114 ceph-mon[33315]: osdmap e668: 8 total, 8 up, 8 in 2024-09-16T09:13:20.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:20 smithi111 ceph-mon[29803]: osdmap e668: 8 total, 8 up, 8 in 2024-09-16T09:13:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:20 smithi111 ceph-mon[40056]: osdmap e668: 8 total, 8 up, 8 in 2024-09-16T09:13:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:21 smithi114 ceph-mon[33315]: osdmap e669: 8 total, 8 up, 8 in 2024-09-16T09:13:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-138","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:21 smithi114 ceph-mon[33315]: pgmap v1024: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:21 smithi111 ceph-mon[29803]: osdmap e669: 8 total, 8 up, 8 in 2024-09-16T09:13:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-138","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:21 smithi111 ceph-mon[29803]: pgmap v1024: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:21 smithi111 ceph-mon[40056]: osdmap e669: 8 total, 8 up, 8 in 2024-09-16T09:13:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-138","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:21.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:21 smithi111 ceph-mon[40056]: pgmap v1024: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:22.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:13:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:13:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:22 smithi114 ceph-mon[33315]: osdmap e670: 8 total, 8 up, 8 in 2024-09-16T09:13:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:22.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:22 smithi111 ceph-mon[29803]: osdmap e670: 8 total, 8 up, 8 in 2024-09-16T09:13:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:22 smithi111 ceph-mon[40056]: osdmap e670: 8 total, 8 up, 8 in 2024-09-16T09:13:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:23.459 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:13:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:13:23.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:23 smithi114 ceph-mon[33315]: osdmap e671: 8 total, 8 up, 8 in 2024-09-16T09:13:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:13:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:23 smithi114 ceph-mon[33315]: pgmap v1027: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:23 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:13:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:23 smithi114 ceph-mon[33315]: osdmap e672: 8 total, 8 up, 8 in 2024-09-16T09:13:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[29803]: osdmap e671: 8 total, 8 up, 8 in 2024-09-16T09:13:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:13:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[29803]: pgmap v1027: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[29803]: osdmap e672: 8 total, 8 up, 8 in 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[40056]: osdmap e671: 8 total, 8 up, 8 in 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[40056]: pgmap v1027: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:13:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:23 smithi111 ceph-mon[40056]: osdmap e672: 8 total, 8 up, 8 in 2024-09-16T09:13:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:13:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:13:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:24 smithi114 ceph-mon[33315]: osdmap e673: 8 total, 8 up, 8 in 2024-09-16T09:13:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:13:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:13:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:13:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:24 smithi111 ceph-mon[29803]: osdmap e673: 8 total, 8 up, 8 in 2024-09-16T09:13:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:13:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:13:24.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:13:24.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:24 smithi111 ceph-mon[40056]: osdmap e673: 8 total, 8 up, 8 in 2024-09-16T09:13:24.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_type","val": "explicit_object"} : dispatch 2024-09-16T09:13:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:25 smithi114 ceph-mon[33315]: pgmap v1030: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:13:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:25 smithi114 ceph-mon[33315]: osdmap e674: 8 total, 8 up, 8 in 2024-09-16T09:13:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:25 smithi111 ceph-mon[29803]: pgmap v1030: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:13:25.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:25 smithi111 ceph-mon[29803]: osdmap e674: 8 total, 8 up, 8 in 2024-09-16T09:13:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:25 smithi111 ceph-mon[40056]: pgmap v1030: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-16T09:13:25.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:25 smithi111 ceph-mon[40056]: osdmap e674: 8 total, 8 up, 8 in 2024-09-16T09:13:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138"} : dispatch 2024-09-16T09:13:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138"} : dispatch 2024-09-16T09:13:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:26.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138"} : dispatch 2024-09-16T09:13:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138"}]': finished 2024-09-16T09:13:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:27 smithi114 ceph-mon[33315]: osdmap e675: 8 total, 8 up, 8 in 2024-09-16T09:13:27.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:27 smithi114 ceph-mon[33315]: pgmap v1033: 204 pgs: 204 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138"}]': finished 2024-09-16T09:13:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:27 smithi111 ceph-mon[29803]: osdmap e675: 8 total, 8 up, 8 in 2024-09-16T09:13:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:27 smithi111 ceph-mon[29803]: pgmap v1033: 204 pgs: 204 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-138"}]': finished 2024-09-16T09:13:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:27 smithi111 ceph-mon[40056]: osdmap e675: 8 total, 8 up, 8 in 2024-09-16T09:13:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:27 smithi111 ceph-mon[40056]: pgmap v1033: 204 pgs: 204 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:28.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:28 smithi114 ceph-mon[33315]: osdmap e676: 8 total, 8 up, 8 in 2024-09-16T09:13:28.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:28 smithi111 ceph-mon[29803]: osdmap e676: 8 total, 8 up, 8 in 2024-09-16T09:13:28.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:28 smithi111 ceph-mon[40056]: osdmap e676: 8 total, 8 up, 8 in 2024-09-16T09:13:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:29 smithi114 ceph-mon[33315]: osdmap e677: 8 total, 8 up, 8 in 2024-09-16T09:13:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-140","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:29 smithi114 ceph-mon[33315]: pgmap v1036: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:29 smithi114 ceph-mon[33315]: osdmap e678: 8 total, 8 up, 8 in 2024-09-16T09:13:29.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[29803]: osdmap e677: 8 total, 8 up, 8 in 2024-09-16T09:13:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-140","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[29803]: pgmap v1036: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[29803]: osdmap e678: 8 total, 8 up, 8 in 2024-09-16T09:13:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[40056]: osdmap e677: 8 total, 8 up, 8 in 2024-09-16T09:13:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-140","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[40056]: pgmap v1036: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[40056]: osdmap e678: 8 total, 8 up, 8 in 2024-09-16T09:13:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:31 smithi114 ceph-mon[33315]: osdmap e679: 8 total, 8 up, 8 in 2024-09-16T09:13:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:31 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_count","val": "3"} : dispatch 2024-09-16T09:13:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:31 smithi114 ceph-mon[33315]: pgmap v1039: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:31.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:13:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:31.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[29803]: osdmap e679: 8 total, 8 up, 8 in 2024-09-16T09:13:31.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_count","val": "3"} : dispatch 2024-09-16T09:13:31.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[29803]: pgmap v1039: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:31.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:13:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[40056]: osdmap e679: 8 total, 8 up, 8 in 2024-09-16T09:13:31.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_count","val": "3"} : dispatch 2024-09-16T09:13:31.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[40056]: pgmap v1039: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:31.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:13:32.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:13:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:13:32.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:32.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:32.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_count","val": "3"}]': finished 2024-09-16T09:13:32.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[40056]: osdmap e680: 8 total, 8 up, 8 in 2024-09-16T09:13:32.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_period","val": "3"} : dispatch 2024-09-16T09:13:32.828 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:32.829 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:32.829 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:32.829 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_count","val": "3"}]': finished 2024-09-16T09:13:32.829 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[29803]: osdmap e680: 8 total, 8 up, 8 in 2024-09-16T09:13:32.829 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_period","val": "3"} : dispatch 2024-09-16T09:13:32.829 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:32 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:13:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:32 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_count","val": "3"}]': finished 2024-09-16T09:13:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:32 smithi114 ceph-mon[33315]: osdmap e680: 8 total, 8 up, 8 in 2024-09-16T09:13:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:32 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_period","val": "3"} : dispatch 2024-09-16T09:13:32.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:33.560 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:13:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:13:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_period","val": "3"}]': finished 2024-09-16T09:13:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:33 smithi114 ceph-mon[33315]: osdmap e681: 8 total, 8 up, 8 in 2024-09-16T09:13:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:13:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:33 smithi114 ceph-mon[33315]: pgmap v1042: 204 pgs: 204 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:33.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:33 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:13:33.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_period","val": "3"}]': finished 2024-09-16T09:13:33.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[29803]: osdmap e681: 8 total, 8 up, 8 in 2024-09-16T09:13:33.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:13:33.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[29803]: pgmap v1042: 204 pgs: 204 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:33.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:13:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_period","val": "3"}]': finished 2024-09-16T09:13:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[40056]: osdmap e681: 8 total, 8 up, 8 in 2024-09-16T09:13:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:13:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[40056]: pgmap v1042: 204 pgs: 204 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:33.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:33 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:13:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:34 smithi114 ceph-mon[33315]: osdmap e682: 8 total, 8 up, 8 in 2024-09-16T09:13:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_fpp","val": ".01"} : dispatch 2024-09-16T09:13:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-16T09:13:34.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:34 smithi114 ceph-mon[33315]: osdmap e683: 8 total, 8 up, 8 in 2024-09-16T09:13:34.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:34 smithi111 ceph-mon[29803]: osdmap e682: 8 total, 8 up, 8 in 2024-09-16T09:13:34.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_fpp","val": ".01"} : dispatch 2024-09-16T09:13:34.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-16T09:13:34.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:34 smithi111 ceph-mon[29803]: osdmap e683: 8 total, 8 up, 8 in 2024-09-16T09:13:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:34 smithi111 ceph-mon[40056]: osdmap e682: 8 total, 8 up, 8 in 2024-09-16T09:13:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_fpp","val": ".01"} : dispatch 2024-09-16T09:13:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-16T09:13:34.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:34 smithi111 ceph-mon[40056]: osdmap e683: 8 total, 8 up, 8 in 2024-09-16T09:13:35.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:35 smithi114 ceph-mon[33315]: pgmap v1045: 204 pgs: 204 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:35.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:35 smithi111 ceph-mon[40056]: pgmap v1045: 204 pgs: 204 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:35.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:35 smithi111 ceph-mon[29803]: pgmap v1045: 204 pgs: 204 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:37 smithi114 ceph-mon[33315]: pgmap v1046: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3.1 KiB/s wr, 2 op/s 2024-09-16T09:13:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:37 smithi111 ceph-mon[29803]: pgmap v1046: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3.1 KiB/s wr, 2 op/s 2024-09-16T09:13:38.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:37 smithi111 ceph-mon[40056]: pgmap v1046: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3.1 KiB/s wr, 2 op/s 2024-09-16T09:13:40.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:39 smithi114 ceph-mon[33315]: pgmap v1047: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 2.6 KiB/s wr, 1 op/s 2024-09-16T09:13:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:39 smithi111 ceph-mon[29803]: pgmap v1047: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 2.6 KiB/s wr, 1 op/s 2024-09-16T09:13:40.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:39 smithi111 ceph-mon[40056]: pgmap v1047: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 2.6 KiB/s wr, 1 op/s 2024-09-16T09:13:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:41 smithi114 ceph-mon[33315]: pgmap v1048: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-09-16T09:13:42.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:41 smithi111 ceph-mon[29803]: pgmap v1048: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-09-16T09:13:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:41 smithi111 ceph-mon[40056]: pgmap v1048: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-09-16T09:13:42.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:13:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:13:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:13:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:13:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:13:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:43 smithi114 ceph-mon[33315]: pgmap v1049: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.3 KiB/s wr, 2 op/s 2024-09-16T09:13:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:43 smithi111 ceph-mon[29803]: pgmap v1049: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.3 KiB/s wr, 2 op/s 2024-09-16T09:13:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:43 smithi111 ceph-mon[40056]: pgmap v1049: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.3 KiB/s wr, 2 op/s 2024-09-16T09:13:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:45 smithi114 ceph-mon[33315]: pgmap v1050: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 5.7 KiB/s wr, 1 op/s 2024-09-16T09:13:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:45 smithi111 ceph-mon[29803]: pgmap v1050: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 5.7 KiB/s wr, 1 op/s 2024-09-16T09:13:46.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:45 smithi111 ceph-mon[40056]: pgmap v1050: 204 pgs: 204 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 5.7 KiB/s wr, 1 op/s 2024-09-16T09:13:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140"} : dispatch 2024-09-16T09:13:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140"} : dispatch 2024-09-16T09:13:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:13:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:13:47.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140"} : dispatch 2024-09-16T09:13:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:47 smithi114 ceph-mon[33315]: pgmap v1051: 204 pgs: 204 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-09-16T09:13:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:47 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140"}]': finished 2024-09-16T09:13:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:47 smithi114 ceph-mon[33315]: osdmap e684: 8 total, 8 up, 8 in 2024-09-16T09:13:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:47 smithi111 ceph-mon[29803]: pgmap v1051: 204 pgs: 204 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-09-16T09:13:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:47 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140"}]': finished 2024-09-16T09:13:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:47 smithi111 ceph-mon[29803]: osdmap e684: 8 total, 8 up, 8 in 2024-09-16T09:13:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:47 smithi111 ceph-mon[40056]: pgmap v1051: 204 pgs: 204 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-09-16T09:13:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:47 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-140"}]': finished 2024-09-16T09:13:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:47 smithi111 ceph-mon[40056]: osdmap e684: 8 total, 8 up, 8 in 2024-09-16T09:13:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:48 smithi114 ceph-mon[33315]: osdmap e685: 8 total, 8 up, 8 in 2024-09-16T09:13:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:48 smithi114 ceph-mon[33315]: osdmap e686: 8 total, 8 up, 8 in 2024-09-16T09:13:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-142","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:48 smithi111 ceph-mon[29803]: osdmap e685: 8 total, 8 up, 8 in 2024-09-16T09:13:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:48 smithi111 ceph-mon[29803]: osdmap e686: 8 total, 8 up, 8 in 2024-09-16T09:13:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-142","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:48 smithi111 ceph-mon[40056]: osdmap e685: 8 total, 8 up, 8 in 2024-09-16T09:13:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:48 smithi111 ceph-mon[40056]: osdmap e686: 8 total, 8 up, 8 in 2024-09-16T09:13:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-142","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:13:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:50 smithi114 ceph-mon[33315]: pgmap v1054: 172 pgs: 172 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:50 smithi114 ceph-mon[33315]: osdmap e687: 8 total, 8 up, 8 in 2024-09-16T09:13:50.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:50 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:50 smithi111 ceph-mon[29803]: pgmap v1054: 172 pgs: 172 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:50 smithi111 ceph-mon[29803]: osdmap e687: 8 total, 8 up, 8 in 2024-09-16T09:13:50.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:50 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:50 smithi111 ceph-mon[40056]: pgmap v1054: 172 pgs: 172 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:13:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:50 smithi111 ceph-mon[40056]: osdmap e687: 8 total, 8 up, 8 in 2024-09-16T09:13:50.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:50 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:13:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:51 smithi114 ceph-mon[33315]: pgmap v1057: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:51 smithi114 ceph-mon[33315]: osdmap e688: 8 total, 8 up, 8 in 2024-09-16T09:13:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-142"} : dispatch 2024-09-16T09:13:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:51 smithi111 ceph-mon[29803]: pgmap v1057: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:51 smithi111 ceph-mon[29803]: osdmap e688: 8 total, 8 up, 8 in 2024-09-16T09:13:52.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-142"} : dispatch 2024-09-16T09:13:52.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:51 smithi111 ceph-mon[40056]: pgmap v1057: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:13:52.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:13:52.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:51 smithi111 ceph-mon[40056]: osdmap e688: 8 total, 8 up, 8 in 2024-09-16T09:13:52.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-142"} : dispatch 2024-09-16T09:13:52.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:13:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:13:53.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-142"}]': finished 2024-09-16T09:13:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:52 smithi114 ceph-mon[33315]: osdmap e689: 8 total, 8 up, 8 in 2024-09-16T09:13:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-142", "mode": "writeback"} : dispatch 2024-09-16T09:13:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:52 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:53.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-142"}]': finished 2024-09-16T09:13:53.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:52 smithi111 ceph-mon[29803]: osdmap e689: 8 total, 8 up, 8 in 2024-09-16T09:13:53.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-142", "mode": "writeback"} : dispatch 2024-09-16T09:13:53.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:52 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:53.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-142"}]': finished 2024-09-16T09:13:53.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:52 smithi111 ceph-mon[40056]: osdmap e689: 8 total, 8 up, 8 in 2024-09-16T09:13:53.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-142", "mode": "writeback"} : dispatch 2024-09-16T09:13:53.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:52 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:13:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:13:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:13:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:53 smithi114 ceph-mon[33315]: pgmap v1060: 204 pgs: 204 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:13:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:53 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:13:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-142", "mode": "writeback"}]': finished 2024-09-16T09:13:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:53 smithi114 ceph-mon[33315]: osdmap e690: 8 total, 8 up, 8 in 2024-09-16T09:13:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:13:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:13:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:53 smithi114 ceph-mon[33315]: osdmap e691: 8 total, 8 up, 8 in 2024-09-16T09:13:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[29803]: pgmap v1060: 204 pgs: 204 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:13:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-142", "mode": "writeback"}]': finished 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[29803]: osdmap e690: 8 total, 8 up, 8 in 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[29803]: osdmap e691: 8 total, 8 up, 8 in 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[40056]: pgmap v1060: 204 pgs: 204 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-142", "mode": "writeback"}]': finished 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[40056]: osdmap e690: 8 total, 8 up, 8 in 2024-09-16T09:13:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:13:54.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:13:54.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:53 smithi111 ceph-mon[40056]: osdmap e691: 8 total, 8 up, 8 in 2024-09-16T09:13:55.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:13:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:13:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:54 smithi114 ceph-mon[33315]: osdmap e692: 8 total, 8 up, 8 in 2024-09-16T09:13:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:13:55.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:13:55.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:13:55.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:54 smithi111 ceph-mon[40056]: osdmap e692: 8 total, 8 up, 8 in 2024-09-16T09:13:55.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:13:55.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:13:55.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:13:55.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:54 smithi111 ceph-mon[29803]: osdmap e692: 8 total, 8 up, 8 in 2024-09-16T09:13:55.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:13:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:55 smithi114 ceph-mon[33315]: pgmap v1063: 204 pgs: 204 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:13:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:55 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:13:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:55 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:13:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:55 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:13:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:55 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:13:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:55 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:13:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:55 smithi114 ceph-mon[33315]: osdmap e693: 8 total, 8 up, 8 in 2024-09-16T09:13:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:13:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[29803]: pgmap v1063: 204 pgs: 204 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:13:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:13:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:13:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:13:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[29803]: osdmap e693: 8 total, 8 up, 8 in 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[40056]: pgmap v1063: 204 pgs: 204 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:13:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[40056]: osdmap e693: 8 total, 8 up, 8 in 2024-09-16T09:13:56.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:13:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:57 smithi114 ceph-mon[33315]: pgmap v1066: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:13:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:57 smithi114 ceph-mon[33315]: osdmap e694: 8 total, 8 up, 8 in 2024-09-16T09:13:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_grade_decay_rate","val": "20"} : dispatch 2024-09-16T09:13:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:57 smithi111 ceph-mon[29803]: pgmap v1066: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:13:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:57 smithi111 ceph-mon[29803]: osdmap e694: 8 total, 8 up, 8 in 2024-09-16T09:13:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_grade_decay_rate","val": "20"} : dispatch 2024-09-16T09:13:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:57 smithi111 ceph-mon[40056]: pgmap v1066: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:13:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:13:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:57 smithi111 ceph-mon[40056]: osdmap e694: 8 total, 8 up, 8 in 2024-09-16T09:13:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_grade_decay_rate","val": "20"} : dispatch 2024-09-16T09:13:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-16T09:13:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:58 smithi114 ceph-mon[33315]: osdmap e695: 8 total, 8 up, 8 in 2024-09-16T09:13:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_search_last_n","val": "1"} : dispatch 2024-09-16T09:13:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-16T09:13:59.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:58 smithi114 ceph-mon[33315]: osdmap e696: 8 total, 8 up, 8 in 2024-09-16T09:13:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-16T09:13:59.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[29803]: osdmap e695: 8 total, 8 up, 8 in 2024-09-16T09:13:59.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_search_last_n","val": "1"} : dispatch 2024-09-16T09:13:59.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-16T09:13:59.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[29803]: osdmap e696: 8 total, 8 up, 8 in 2024-09-16T09:13:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-16T09:13:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[40056]: osdmap e695: 8 total, 8 up, 8 in 2024-09-16T09:13:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_search_last_n","val": "1"} : dispatch 2024-09-16T09:13:59.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-16T09:13:59.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:58 smithi111 ceph-mon[40056]: osdmap e696: 8 total, 8 up, 8 in 2024-09-16T09:14:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:59 smithi114 ceph-mon[33315]: pgmap v1069: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:14:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:13:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:59 smithi111 ceph-mon[29803]: pgmap v1069: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:14:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:13:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:00.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:59 smithi111 ceph-mon[40056]: pgmap v1069: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:14:00.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:13:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:01.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:00 smithi114 ceph-mon[33315]: osdmap e697: 8 total, 8 up, 8 in 2024-09-16T09:14:01.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142"} : dispatch 2024-09-16T09:14:01.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:01.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:00 smithi111 ceph-mon[29803]: osdmap e697: 8 total, 8 up, 8 in 2024-09-16T09:14:01.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142"} : dispatch 2024-09-16T09:14:01.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:01.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:00 smithi111 ceph-mon[40056]: osdmap e697: 8 total, 8 up, 8 in 2024-09-16T09:14:01.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142"} : dispatch 2024-09-16T09:14:02.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:01 smithi114 ceph-mon[33315]: pgmap v1072: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:14:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142"}]': finished 2024-09-16T09:14:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:01 smithi114 ceph-mon[33315]: osdmap e698: 8 total, 8 up, 8 in 2024-09-16T09:14:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142"} : dispatch 2024-09-16T09:14:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[29803]: pgmap v1072: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:14:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142"}]': finished 2024-09-16T09:14:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[29803]: osdmap e698: 8 total, 8 up, 8 in 2024-09-16T09:14:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142"} : dispatch 2024-09-16T09:14:02.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:02.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[40056]: pgmap v1072: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:14:02.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142"}]': finished 2024-09-16T09:14:02.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[40056]: osdmap e698: 8 total, 8 up, 8 in 2024-09-16T09:14:02.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:02.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-142"} : dispatch 2024-09-16T09:14:02.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:02.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:14:02 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:14:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:02 smithi114 ceph-mon[33315]: osdmap e699: 8 total, 8 up, 8 in 2024-09-16T09:14:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:02 smithi114 ceph-mon[33315]: osdmap e700: 8 total, 8 up, 8 in 2024-09-16T09:14:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-144","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:02 smithi111 ceph-mon[29803]: osdmap e699: 8 total, 8 up, 8 in 2024-09-16T09:14:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:02 smithi111 ceph-mon[29803]: osdmap e700: 8 total, 8 up, 8 in 2024-09-16T09:14:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-144","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:02 smithi111 ceph-mon[40056]: osdmap e699: 8 total, 8 up, 8 in 2024-09-16T09:14:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:02 smithi111 ceph-mon[40056]: osdmap e700: 8 total, 8 up, 8 in 2024-09-16T09:14:03.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-144","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:14:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:14:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:14:04.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:03 smithi114 ceph-mon[33315]: pgmap v1075: 172 pgs: 172 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:14:04.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:04.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:03 smithi114 ceph-mon[33315]: osdmap e701: 8 total, 8 up, 8 in 2024-09-16T09:14:04.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:14:04.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:03 smithi111 ceph-mon[29803]: pgmap v1075: 172 pgs: 172 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:14:04.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:04.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:03 smithi111 ceph-mon[29803]: osdmap e701: 8 total, 8 up, 8 in 2024-09-16T09:14:04.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:14:04.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:03 smithi111 ceph-mon[40056]: pgmap v1075: 172 pgs: 172 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:14:04.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:04.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:03 smithi111 ceph-mon[40056]: osdmap e701: 8 total, 8 up, 8 in 2024-09-16T09:14:04.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:14:06.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:05 smithi114 ceph-mon[33315]: pgmap v1078: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:14:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:14:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:05 smithi114 ceph-mon[33315]: osdmap e702: 8 total, 8 up, 8 in 2024-09-16T09:14:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:05 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-144"} : dispatch 2024-09-16T09:14:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:05 smithi111 ceph-mon[29803]: pgmap v1078: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:14:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:14:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:05 smithi111 ceph-mon[29803]: osdmap e702: 8 total, 8 up, 8 in 2024-09-16T09:14:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:05 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-144"} : dispatch 2024-09-16T09:14:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:05 smithi111 ceph-mon[40056]: pgmap v1078: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-16T09:14:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:14:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:05 smithi111 ceph-mon[40056]: osdmap e702: 8 total, 8 up, 8 in 2024-09-16T09:14:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:05 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-144"} : dispatch 2024-09-16T09:14:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-144"}]': finished 2024-09-16T09:14:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:06 smithi114 ceph-mon[33315]: osdmap e703: 8 total, 8 up, 8 in 2024-09-16T09:14:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-144", "mode": "readproxy"} : dispatch 2024-09-16T09:14:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:06 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:14:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:06 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-144", "mode": "readproxy"}]': finished 2024-09-16T09:14:07.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:06 smithi114 ceph-mon[33315]: osdmap e704: 8 total, 8 up, 8 in 2024-09-16T09:14:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-144"}]': finished 2024-09-16T09:14:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[29803]: osdmap e703: 8 total, 8 up, 8 in 2024-09-16T09:14:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-144", "mode": "readproxy"} : dispatch 2024-09-16T09:14:07.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:14:07.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-144", "mode": "readproxy"}]': finished 2024-09-16T09:14:07.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[29803]: osdmap e704: 8 total, 8 up, 8 in 2024-09-16T09:14:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-144"}]': finished 2024-09-16T09:14:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[40056]: osdmap e703: 8 total, 8 up, 8 in 2024-09-16T09:14:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-144", "mode": "readproxy"} : dispatch 2024-09-16T09:14:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:14:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-144", "mode": "readproxy"}]': finished 2024-09-16T09:14:07.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:06 smithi111 ceph-mon[40056]: osdmap e704: 8 total, 8 up, 8 in 2024-09-16T09:14:08.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:07 smithi114 ceph-mon[33315]: pgmap v1081: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:14:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:07 smithi111 ceph-mon[29803]: pgmap v1081: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:14:08.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:07 smithi111 ceph-mon[40056]: pgmap v1081: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-16T09:14:10.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:09 smithi114 ceph-mon[33315]: pgmap v1083: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 210 B/s wr, 1 op/s 2024-09-16T09:14:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:09 smithi111 ceph-mon[29803]: pgmap v1083: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 210 B/s wr, 1 op/s 2024-09-16T09:14:10.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:09 smithi111 ceph-mon[40056]: pgmap v1083: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 210 B/s wr, 1 op/s 2024-09-16T09:14:12.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:11 smithi114 ceph-mon[33315]: pgmap v1084: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-09-16T09:14:12.206 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:11 smithi111 ceph-mon[29803]: pgmap v1084: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-09-16T09:14:12.207 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:11 smithi111 ceph-mon[40056]: pgmap v1084: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-09-16T09:14:12.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:14:12 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:14:13.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:12 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:12 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:13.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:12 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:13.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:14:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:14:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:14:14.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:13 smithi114 ceph-mon[33315]: pgmap v1085: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1 op/s 2024-09-16T09:14:14.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:13 smithi111 ceph-mon[29803]: pgmap v1085: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1 op/s 2024-09-16T09:14:14.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:13 smithi111 ceph-mon[40056]: pgmap v1085: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1 op/s 2024-09-16T09:14:16.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:15 smithi114 ceph-mon[33315]: pgmap v1086: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2024-09-16T09:14:16.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:15 smithi111 ceph-mon[29803]: pgmap v1086: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2024-09-16T09:14:16.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:15 smithi111 ceph-mon[40056]: pgmap v1086: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2024-09-16T09:14:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:17 smithi114 ceph-mon[33315]: pgmap v1087: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T09:14:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:17.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:17.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:17 smithi111 ceph-mon[29803]: pgmap v1087: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T09:14:17.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:17.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:17.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:17 smithi111 ceph-mon[40056]: pgmap v1087: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T09:14:17.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:18 smithi114 ceph-mon[33315]: osdmap e705: 8 total, 8 up, 8 in 2024-09-16T09:14:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144"} : dispatch 2024-09-16T09:14:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:18 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:14:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144"}]': finished 2024-09-16T09:14:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:18 smithi114 ceph-mon[33315]: osdmap e706: 8 total, 8 up, 8 in 2024-09-16T09:14:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[29803]: osdmap e705: 8 total, 8 up, 8 in 2024-09-16T09:14:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144"} : dispatch 2024-09-16T09:14:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:14:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144"}]': finished 2024-09-16T09:14:18.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[29803]: osdmap e706: 8 total, 8 up, 8 in 2024-09-16T09:14:18.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:18.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[40056]: osdmap e705: 8 total, 8 up, 8 in 2024-09-16T09:14:18.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144"} : dispatch 2024-09-16T09:14:18.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:14:18.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144"}]': finished 2024-09-16T09:14:18.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:18 smithi111 ceph-mon[40056]: osdmap e706: 8 total, 8 up, 8 in 2024-09-16T09:14:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144"} : dispatch 2024-09-16T09:14:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:19 smithi114 ceph-mon[33315]: pgmap v1090: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-16T09:14:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144"} : dispatch 2024-09-16T09:14:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:19 smithi111 ceph-mon[29803]: pgmap v1090: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-16T09:14:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-144"} : dispatch 2024-09-16T09:14:19.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:19 smithi111 ceph-mon[40056]: pgmap v1090: 204 pgs: 204 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-16T09:14:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:20 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:20 smithi114 ceph-mon[33315]: osdmap e707: 8 total, 8 up, 8 in 2024-09-16T09:14:20.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:20 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:20.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:20 smithi111 ceph-mon[29803]: osdmap e707: 8 total, 8 up, 8 in 2024-09-16T09:14:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:20 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:20 smithi111 ceph-mon[40056]: osdmap e707: 8 total, 8 up, 8 in 2024-09-16T09:14:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:21 smithi114 ceph-mon[33315]: osdmap e708: 8 total, 8 up, 8 in 2024-09-16T09:14:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-146","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:21 smithi114 ceph-mon[33315]: pgmap v1093: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:14:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:21 smithi111 ceph-mon[40056]: osdmap e708: 8 total, 8 up, 8 in 2024-09-16T09:14:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-146","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:21 smithi111 ceph-mon[40056]: pgmap v1093: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:14:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:21 smithi111 ceph-mon[29803]: osdmap e708: 8 total, 8 up, 8 in 2024-09-16T09:14:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-146","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:21 smithi111 ceph-mon[29803]: pgmap v1093: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail 2024-09-16T09:14:22.636 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:14:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:14:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:22 smithi114 ceph-mon[33315]: osdmap e709: 8 total, 8 up, 8 in 2024-09-16T09:14:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:14:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:22 smithi111 ceph-mon[29803]: osdmap e709: 8 total, 8 up, 8 in 2024-09-16T09:14:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:14:22.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:22 smithi111 ceph-mon[40056]: osdmap e709: 8 total, 8 up, 8 in 2024-09-16T09:14:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:14:22.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:23.639 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:14:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:14:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:14:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:23 smithi114 ceph-mon[33315]: osdmap e710: 8 total, 8 up, 8 in 2024-09-16T09:14:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-146"} : dispatch 2024-09-16T09:14:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:23 smithi114 ceph-mon[33315]: pgmap v1096: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:14:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-146"}]': finished 2024-09-16T09:14:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:23 smithi114 ceph-mon[33315]: osdmap e711: 8 total, 8 up, 8 in 2024-09-16T09:14:23.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:23 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-146", "mode": "writeback"} : dispatch 2024-09-16T09:14:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:14:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[29803]: osdmap e710: 8 total, 8 up, 8 in 2024-09-16T09:14:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-146"} : dispatch 2024-09-16T09:14:23.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[29803]: pgmap v1096: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:14:23.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-146"}]': finished 2024-09-16T09:14:23.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[29803]: osdmap e711: 8 total, 8 up, 8 in 2024-09-16T09:14:23.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-146", "mode": "writeback"} : dispatch 2024-09-16T09:14:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:14:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[40056]: osdmap e710: 8 total, 8 up, 8 in 2024-09-16T09:14:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-146"} : dispatch 2024-09-16T09:14:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[40056]: pgmap v1096: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:14:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi111-76716-111", "overlaypool": "test-rados-api-smithi111-76716-146"}]': finished 2024-09-16T09:14:23.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[40056]: osdmap e711: 8 total, 8 up, 8 in 2024-09-16T09:14:23.928 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:23 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-146", "mode": "writeback"} : dispatch 2024-09-16T09:14:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:24 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:14:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:24 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-146", "mode": "writeback"}]': finished 2024-09-16T09:14:24.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:24 smithi114 ceph-mon[33315]: osdmap e712: 8 total, 8 up, 8 in 2024-09-16T09:14:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:24 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:14:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:24 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-146", "mode": "writeback"}]': finished 2024-09-16T09:14:24.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:24 smithi111 ceph-mon[29803]: osdmap e712: 8 total, 8 up, 8 in 2024-09-16T09:14:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:24 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T09:14:24.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:24 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi111-76716-146", "mode": "writeback"}]': finished 2024-09-16T09:14:24.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:24 smithi111 ceph-mon[40056]: osdmap e712: 8 total, 8 up, 8 in 2024-09-16T09:14:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:25 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:14:25.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:25 smithi114 ceph-mon[33315]: pgmap v1099: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:14:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:25 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:14:25.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:25 smithi111 ceph-mon[29803]: pgmap v1099: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:14:25.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:25 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_count","val": "2"} : dispatch 2024-09-16T09:14:25.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:25 smithi111 ceph-mon[40056]: pgmap v1099: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:14:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:14:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:26 smithi114 ceph-mon[33315]: osdmap e713: 8 total, 8 up, 8 in 2024-09-16T09:14:26.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:26 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:14:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:14:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:26 smithi111 ceph-mon[29803]: osdmap e713: 8 total, 8 up, 8 in 2024-09-16T09:14:26.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:26 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:14:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_count","val": "2"}]': finished 2024-09-16T09:14:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:26 smithi111 ceph-mon[40056]: osdmap e713: 8 total, 8 up, 8 in 2024-09-16T09:14:26.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:26 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_period","val": "600"} : dispatch 2024-09-16T09:14:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:14:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[29803]: osdmap e714: 8 total, 8 up, 8 in 2024-09-16T09:14:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:14:27.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[29803]: pgmap v1102: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T09:14:27.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:14:27.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:14:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:14:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[40056]: osdmap e714: 8 total, 8 up, 8 in 2024-09-16T09:14:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:14:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[40056]: pgmap v1102: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T09:14:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:14:27.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:27 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:14:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_period","val": "600"}]': finished 2024-09-16T09:14:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:27 smithi114 ceph-mon[33315]: osdmap e714: 8 total, 8 up, 8 in 2024-09-16T09:14:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_type","val": "bloom"} : dispatch 2024-09-16T09:14:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:27 smithi114 ceph-mon[33315]: pgmap v1102: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T09:14:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:27 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-16T09:14:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:27 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "hit_set_type","val": "bloom"}]': finished 2024-09-16T09:14:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[29803]: osdmap e715: 8 total, 8 up, 8 in 2024-09-16T09:14:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:14:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:28.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:14:28.930 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[29803]: osdmap e716: 8 total, 8 up, 8 in 2024-09-16T09:14:28.930 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "target_max_objects","val": "1"} : dispatch 2024-09-16T09:14:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[40056]: osdmap e715: 8 total, 8 up, 8 in 2024-09-16T09:14:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:14:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:14:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[40056]: osdmap e716: 8 total, 8 up, 8 in 2024-09-16T09:14:28.930 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:28 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "target_max_objects","val": "1"} : dispatch 2024-09-16T09:14:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:28 smithi114 ceph-mon[33315]: osdmap e715: 8 total, 8 up, 8 in 2024-09-16T09:14:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "min_read_recency_for_promote","val": "1"} : dispatch 2024-09-16T09:14:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:28 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-16T09:14:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:28 smithi114 ceph-mon[33315]: osdmap e716: 8 total, 8 up, 8 in 2024-09-16T09:14:29.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:28 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "target_max_objects","val": "1"} : dispatch 2024-09-16T09:14:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:29 smithi111 ceph-mon[29803]: pgmap v1105: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T09:14:29.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:29 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "target_max_objects","val": "1"}]': finished 2024-09-16T09:14:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:29 smithi111 ceph-mon[40056]: pgmap v1105: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T09:14:29.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:29 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "target_max_objects","val": "1"}]': finished 2024-09-16T09:14:30.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:29 smithi114 ceph-mon[33315]: pgmap v1105: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-16T09:14:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:29 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-146","var": "target_max_objects","val": "1"}]': finished 2024-09-16T09:14:30.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:30 smithi111 ceph-mon[29803]: osdmap e717: 8 total, 8 up, 8 in 2024-09-16T09:14:30.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:30 smithi111 ceph-mon[40056]: osdmap e717: 8 total, 8 up, 8 in 2024-09-16T09:14:31.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:30 smithi114 ceph-mon[33315]: osdmap e717: 8 total, 8 up, 8 in 2024-09-16T09:14:32.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:31 smithi114 ceph-mon[33315]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:14:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:31 smithi111 ceph-mon[29803]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:14:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:32.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:31 smithi111 ceph-mon[40056]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:14:32.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:32.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:32.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:14:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:14:33.077 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:32 smithi111 ceph-mon[29803]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-16T09:14:33.077 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:33.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:32 smithi111 ceph-mon[40056]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-16T09:14:33.077 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:33.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:32 smithi114 ceph-mon[33315]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-16T09:14:33.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:33.427 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:14:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:14:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:14:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:33 smithi114 ceph-mon[33315]: pgmap v1108: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:33 smithi111 ceph-mon[29803]: pgmap v1108: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:34.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:33 smithi111 ceph-mon[40056]: pgmap v1108: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:35 smithi114 ceph-mon[33315]: pgmap v1109: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 723 B/s rd, 0 B/s wr, 0 op/s 2024-09-16T09:14:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:35 smithi111 ceph-mon[29803]: pgmap v1109: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 723 B/s rd, 0 B/s wr, 0 op/s 2024-09-16T09:14:36.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:35 smithi111 ceph-mon[40056]: pgmap v1109: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 723 B/s rd, 0 B/s wr, 0 op/s 2024-09-16T09:14:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:37 smithi114 ceph-mon[33315]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:37 smithi111 ceph-mon[29803]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:37 smithi111 ceph-mon[40056]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:39 smithi114 ceph-mon[33315]: pgmap v1111: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:39 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:39 smithi111 ceph-mon[29803]: pgmap v1111: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:39 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:40.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:39 smithi111 ceph-mon[40056]: pgmap v1111: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:40.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:39 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:41.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:41.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:40 smithi114 ceph-mon[33315]: osdmap e718: 8 total, 8 up, 8 in 2024-09-16T09:14:41.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146"} : dispatch 2024-09-16T09:14:41.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:41.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:40 smithi111 ceph-mon[29803]: osdmap e718: 8 total, 8 up, 8 in 2024-09-16T09:14:41.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146"} : dispatch 2024-09-16T09:14:41.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:41.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:40 smithi111 ceph-mon[40056]: osdmap e718: 8 total, 8 up, 8 in 2024-09-16T09:14:41.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146"} : dispatch 2024-09-16T09:14:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:41 smithi114 ceph-mon[33315]: pgmap v1113: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146"}]': finished 2024-09-16T09:14:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:41 smithi114 ceph-mon[33315]: osdmap e719: 8 total, 8 up, 8 in 2024-09-16T09:14:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146"} : dispatch 2024-09-16T09:14:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[29803]: pgmap v1113: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146"}]': finished 2024-09-16T09:14:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[29803]: osdmap e719: 8 total, 8 up, 8 in 2024-09-16T09:14:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146"} : dispatch 2024-09-16T09:14:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[40056]: pgmap v1113: 204 pgs: 204 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-16T09:14:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146"}]': finished 2024-09-16T09:14:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[40056]: osdmap e719: 8 total, 8 up, 8 in 2024-09-16T09:14:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:42.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-146"} : dispatch 2024-09-16T09:14:42.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:14:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:14:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:42 smithi114 ceph-mon[33315]: osdmap e720: 8 total, 8 up, 8 in 2024-09-16T09:14:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:42 smithi111 ceph-mon[29803]: osdmap e720: 8 total, 8 up, 8 in 2024-09-16T09:14:43.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:43.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:42 smithi111 ceph-mon[40056]: osdmap e720: 8 total, 8 up, 8 in 2024-09-16T09:14:43.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:14:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:14:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:14:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:43 smithi114 ceph-mon[33315]: pgmap v1116: 172 pgs: 172 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-16T09:14:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:43 smithi114 ceph-mon[33315]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-16T09:14:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:43 smithi114 ceph-mon[33315]: osdmap e721: 8 total, 8 up, 8 in 2024-09-16T09:14:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:43 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-148","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:43 smithi111 ceph-mon[29803]: pgmap v1116: 172 pgs: 172 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-16T09:14:44.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:43 smithi111 ceph-mon[29803]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-16T09:14:44.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:43 smithi111 ceph-mon[29803]: osdmap e721: 8 total, 8 up, 8 in 2024-09-16T09:14:44.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:43 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-148","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:43 smithi111 ceph-mon[40056]: pgmap v1116: 172 pgs: 172 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-16T09:14:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:43 smithi111 ceph-mon[40056]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-16T09:14:44.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:43 smithi111 ceph-mon[40056]: osdmap e721: 8 total, 8 up, 8 in 2024-09-16T09:14:44.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:43 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-148","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:45.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:45.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:44 smithi114 ceph-mon[33315]: osdmap e722: 8 total, 8 up, 8 in 2024-09-16T09:14:45.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:44 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:14:45.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:45.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:44 smithi111 ceph-mon[29803]: osdmap e722: 8 total, 8 up, 8 in 2024-09-16T09:14:45.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:44 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:14:45.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:45.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:44 smithi111 ceph-mon[40056]: osdmap e722: 8 total, 8 up, 8 in 2024-09-16T09:14:45.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:44 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148", "force_nonempty": "--force-nonempty" } : dispatch 2024-09-16T09:14:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:45 smithi114 ceph-mon[33315]: pgmap v1119: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-16T09:14:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:14:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:45 smithi114 ceph-mon[33315]: osdmap e723: 8 total, 8 up, 8 in 2024-09-16T09:14:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:45 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148"} : dispatch 2024-09-16T09:14:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:45 smithi111 ceph-mon[29803]: pgmap v1119: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-16T09:14:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:14:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:45 smithi111 ceph-mon[29803]: osdmap e723: 8 total, 8 up, 8 in 2024-09-16T09:14:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:45 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148"} : dispatch 2024-09-16T09:14:46.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:45 smithi111 ceph-mon[40056]: pgmap v1119: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 769 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-16T09:14:46.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148", "force_nonempty": "--force-nonempty" }]': finished 2024-09-16T09:14:46.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:45 smithi111 ceph-mon[40056]: osdmap e723: 8 total, 8 up, 8 in 2024-09-16T09:14:46.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:45 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148"} : dispatch 2024-09-16T09:14:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148"}]': finished 2024-09-16T09:14:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:46 smithi114 ceph-mon[33315]: osdmap e724: 8 total, 8 up, 8 in 2024-09-16T09:14:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:46 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148"} : dispatch 2024-09-16T09:14:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148"}]': finished 2024-09-16T09:14:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[29803]: osdmap e724: 8 total, 8 up, 8 in 2024-09-16T09:14:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148"} : dispatch 2024-09-16T09:14:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148"}]': finished 2024-09-16T09:14:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[40056]: osdmap e724: 8 total, 8 up, 8 in 2024-09-16T09:14:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-148"} : dispatch 2024-09-16T09:14:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:14:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:47 smithi114 ceph-mon[33315]: pgmap v1122: 204 pgs: 204 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:14:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:47 smithi114 ceph-mon[33315]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:47 smithi114 ceph-mon[33315]: osdmap e725: 8 total, 8 up, 8 in 2024-09-16T09:14:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:47 smithi111 ceph-mon[29803]: pgmap v1122: 204 pgs: 204 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:14:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:47 smithi111 ceph-mon[29803]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:47 smithi111 ceph-mon[29803]: osdmap e725: 8 total, 8 up, 8 in 2024-09-16T09:14:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:47 smithi111 ceph-mon[40056]: pgmap v1122: 204 pgs: 204 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-16T09:14:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:47 smithi111 ceph-mon[40056]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:48.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:47 smithi111 ceph-mon[40056]: osdmap e725: 8 total, 8 up, 8 in 2024-09-16T09:14:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:48 smithi114 ceph-mon[33315]: osdmap e726: 8 total, 8 up, 8 in 2024-09-16T09:14:49.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:48 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-150","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:48 smithi111 ceph-mon[29803]: osdmap e726: 8 total, 8 up, 8 in 2024-09-16T09:14:49.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:48 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-150","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:48 smithi111 ceph-mon[40056]: osdmap e726: 8 total, 8 up, 8 in 2024-09-16T09:14:49.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:48 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-150","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:50.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:49 smithi114 ceph-mon[33315]: pgmap v1125: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:14:50.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:50.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:49 smithi114 ceph-mon[33315]: osdmap e727: 8 total, 8 up, 8 in 2024-09-16T09:14:50.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:50.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:49 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-150"} : dispatch 2024-09-16T09:14:50.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:49 smithi114 ceph-mon[33315]: osdmap e728: 8 total, 8 up, 8 in 2024-09-16T09:14:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[29803]: pgmap v1125: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:14:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[29803]: osdmap e727: 8 total, 8 up, 8 in 2024-09-16T09:14:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-150"} : dispatch 2024-09-16T09:14:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[29803]: osdmap e728: 8 total, 8 up, 8 in 2024-09-16T09:14:50.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[40056]: pgmap v1125: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-16T09:14:50.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:50.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[40056]: osdmap e727: 8 total, 8 up, 8 in 2024-09-16T09:14:50.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:50.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-150"} : dispatch 2024-09-16T09:14:50.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:49 smithi111 ceph-mon[40056]: osdmap e728: 8 total, 8 up, 8 in 2024-09-16T09:14:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:51 smithi114 ceph-mon[33315]: pgmap v1128: 172 pgs: 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:14:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:51 smithi114 ceph-mon[33315]: osdmap e729: 8 total, 8 up, 8 in 2024-09-16T09:14:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:51 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-152","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:51 smithi111 ceph-mon[29803]: pgmap v1128: 172 pgs: 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:14:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:51 smithi111 ceph-mon[29803]: osdmap e729: 8 total, 8 up, 8 in 2024-09-16T09:14:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:51 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-152","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:52.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:51 smithi111 ceph-mon[40056]: pgmap v1128: 172 pgs: 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:14:52.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:51 smithi111 ceph-mon[40056]: osdmap e729: 8 total, 8 up, 8 in 2024-09-16T09:14:52.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:51 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-152","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:52.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:14:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:14:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:52 smithi114 ceph-mon[33315]: osdmap e730: 8 total, 8 up, 8 in 2024-09-16T09:14:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-152"} : dispatch 2024-09-16T09:14:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:52 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[40056]: osdmap e730: 8 total, 8 up, 8 in 2024-09-16T09:14:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-152"} : dispatch 2024-09-16T09:14:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:53.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:53.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[29803]: osdmap e730: 8 total, 8 up, 8 in 2024-09-16T09:14:53.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:53.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-152"} : dispatch 2024-09-16T09:14:53.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:52 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:14:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:14:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:14:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:14:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:53 smithi114 ceph-mon[33315]: pgmap v1131: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:14:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:53 smithi114 ceph-mon[33315]: osdmap e731: 8 total, 8 up, 8 in 2024-09-16T09:14:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:53 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:53 smithi111 ceph-mon[29803]: pgmap v1131: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:14:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:53 smithi111 ceph-mon[29803]: osdmap e731: 8 total, 8 up, 8 in 2024-09-16T09:14:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:53 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:53 smithi111 ceph-mon[40056]: pgmap v1131: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:14:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:53 smithi111 ceph-mon[40056]: osdmap e731: 8 total, 8 up, 8 in 2024-09-16T09:14:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:53 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:14:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:54 smithi114 ceph-mon[33315]: osdmap e732: 8 total, 8 up, 8 in 2024-09-16T09:14:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-154","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:55.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:54 smithi111 ceph-mon[29803]: osdmap e732: 8 total, 8 up, 8 in 2024-09-16T09:14:55.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-154","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:55.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:54 smithi111 ceph-mon[40056]: osdmap e732: 8 total, 8 up, 8 in 2024-09-16T09:14:55.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-154","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-16T09:14:56.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:55 smithi114 ceph-mon[33315]: pgmap v1134: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:14:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:55 smithi114 ceph-mon[33315]: osdmap e733: 8 total, 8 up, 8 in 2024-09-16T09:14:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-111","var": "dedup_tier","val": "test-rados-api-smithi111-76716-154"} : dispatch 2024-09-16T09:14:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:55 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-154"} : dispatch 2024-09-16T09:14:56.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:55 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:14:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[29803]: pgmap v1134: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:14:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[29803]: osdmap e733: 8 total, 8 up, 8 in 2024-09-16T09:14:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-111","var": "dedup_tier","val": "test-rados-api-smithi111-76716-154"} : dispatch 2024-09-16T09:14:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-154"} : dispatch 2024-09-16T09:14:56.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:14:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[40056]: pgmap v1134: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-16T09:14:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi111-76716-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T09:14:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[40056]: osdmap e733: 8 total, 8 up, 8 in 2024-09-16T09:14:56.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd pool set","pool":"test-rados-api-smithi111-76716-111","var": "dedup_tier","val": "test-rados-api-smithi111-76716-154"} : dispatch 2024-09-16T09:14:56.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:56.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-rados-api-smithi111-76716-111", "tierpool": "test-rados-api-smithi111-76716-154"} : dispatch 2024-09-16T09:14:56.178 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:55 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:14:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:56 smithi114 ceph-mon[33315]: osdmap e734: 8 total, 8 up, 8 in 2024-09-16T09:14:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:56 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:56 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:56 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:56 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:56 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:14:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:56 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:14:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:56 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[29803]: osdmap e734: 8 total, 8 up, 8 in 2024-09-16T09:14:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[40056]: osdmap e734: 8 total, 8 up, 8 in 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:14:57.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:56 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:14:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:57 smithi114 ceph-mon[33315]: pgmap v1137: 172 pgs: 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-16T09:14:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:57 smithi114 ceph-mon[33315]: osdmap e735: 8 total, 8 up, 8 in 2024-09-16T09:14:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:57 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:57 smithi111 ceph-mon[40056]: pgmap v1137: 172 pgs: 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-16T09:14:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:57 smithi111 ceph-mon[40056]: osdmap e735: 8 total, 8 up, 8 in 2024-09-16T09:14:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:57 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:57 smithi111 ceph-mon[29803]: pgmap v1137: 172 pgs: 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-16T09:14:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:57 smithi111 ceph-mon[29803]: osdmap e735: 8 total, 8 up, 8 in 2024-09-16T09:14:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:57 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:58.922 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7819 ms) 2024-09-16T09:14:58.922 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-16T09:14:58.922 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13074 ms) 2024-09-16T09:14:58.922 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-16T09:14:58.922 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8063 ms) 2024-09-16T09:14:58.922 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-16T09:14:58.922 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13064 ms) 2024-09-16T09:14:58.922 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-16T09:14:58.922 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7824 ms) 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8053 ms) 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: ok, hit_set contains 336:602f83fe:::foo:head 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8103 ms) 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,0 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: first is 1726478015 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,0 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,0 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,1726478017,1726478018,0 2024-09-16T09:14:58.923 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,1726478017,1726478018,0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,1726478017,1726478018,0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,1726478017,1726478018,1726478020,1726478021,0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,1726478017,1726478018,1726478020,1726478021,0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,1726478017,1726478018,1726478020,1726478021,0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,1726478017,1726478018,1726478020,1726478021,1726478023,1726478024,0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,1726478017,1726478018,1726478020,1726478021,1726478023,1726478024,0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478015,1726478017,1726478018,1726478020,1726478021,1726478023,1726478024,0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: got ls 1726478018,1726478020,1726478021,1726478023,1726478024,1726478026,1726478027,0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: first now 1726478018, trimmed 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20255 ms) 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: foo0 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T09:14:58.924 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14057 ms) 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17762 ms) 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22200 ms) 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5037 ms) 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3010 ms) 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3023 ms) 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3019 ms) 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (229816 ms total) 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: 2024-09-16T09:14:58.925 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-16T09:14:58.926 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (843965 ms total) 2024-09-16T09:14:58.926 INFO:tasks.workunit.client.0.smithi111.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-16T09:14:58.930 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.930 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76403 2024-09-16T09:14:58.930 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76403 2024-09-16T09:14:58.930 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.930 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76566 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76566 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77137 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77137 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76918 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76918 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76729 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76729 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76475 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76475 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76794 2024-09-16T09:14:58.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76794 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77276 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77276 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77295 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77295 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76501 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76501 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77177 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77177 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76378 2024-09-16T09:14:58.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76378 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76440 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76440 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77090 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77090 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76369 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76369 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76390 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76390 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77536 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77536 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76620 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76620 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77678 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77678 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76768 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76768 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77224 2024-09-16T09:14:58.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77224 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77720 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77720 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76698 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76698 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77123 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77123 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77456 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77456 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77197 2024-09-16T09:14:58.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77197 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77630 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77630 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77066 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77066 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76536 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76536 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77697 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77697 2024-09-16T09:14:58.936 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76971 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76971 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76362 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76362 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76420 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76420 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76855 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76855 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76460 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76460 2024-09-16T09:14:58.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=76993 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 76993 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77030 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77030 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77374 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77374 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ for t in "${!pids[@]}" 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=77761 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 77761 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ exit 0 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ cleanup 2024-09-16T09:14:58.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ pkill -P 76356 2024-09-16T09:14:58.946 INFO:tasks.workunit.client.0.smithi111.stderr:+ true 2024-09-16T09:14:58.946 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T09:14:58.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T09:14:58.976 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-09-16T09:14:58.978 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-09-16T09:14:58.978 DEBUG:teuthology.orchestra.run.smithi111:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8293d73f8690540e843a81caec373f9cc29cf705 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-09-16T09:14:59.038 INFO:tasks.workunit.client.0.smithi111.stderr:++ uuidgen 2024-09-16T09:14:59.048 INFO:tasks.workunit.client.0.smithi111.stderr:+ p=aa242ffd-5c69-42f2-92b2-30d0602f301c 2024-09-16T09:14:59.048 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool create aa242ffd-5c69-42f2-92b2-30d0602f301c 12 2024-09-16T09:14:59.134 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.130+0000 7f2b39a6a640 1 -- 172.21.15.111:0/3290282884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b341043b0 msgr2=0x7f2b34104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:14:59.134 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.130+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/3290282884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b341043b0 0x7f2b34104810 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009900 tx=0x7f2b1c02a410 comp rx=0 tx=0).stop 2024-09-16T09:14:59.134 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.130+0000 7f2b39a6a640 1 -- 172.21.15.111:0/3290282884 shutdown_connections 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.130+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/3290282884 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b34104d50 0x7f2b3410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.130+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/3290282884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b341043b0 0x7f2b34104810 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.130+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/3290282884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b34101f90 0x7f2b34102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.130+0000 7f2b39a6a640 1 -- 172.21.15.111:0/3290282884 >> 172.21.15.111:0/3290282884 conn(0x7f2b340776a0 msgr2=0x7f2b34077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 -- 172.21.15.111:0/3290282884 shutdown_connections 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 -- 172.21.15.111:0/3290282884 wait complete. 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 Processor -- start 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 -- start start 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b34101f90 0x7f2b341995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:14:59.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b341043b0 0x7f2b34199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:14:59.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b34104d50 0x7f2b3419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:14:59.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2b341110e0 con 0x7f2b34101f90 2024-09-16T09:14:59.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b34110f60 con 0x7f2b341043b0 2024-09-16T09:14:59.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.131+0000 7f2b39a6a640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f2b34111260 con 0x7f2b34104d50 2024-09-16T09:14:59.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b337fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b34101f90 0x7f2b341995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:14:59.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b33fff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b34104d50 0x7f2b3419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:14:59.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b337fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b34101f90 0x7f2b341995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49686/0 (socket says 172.21.15.111:49686) 2024-09-16T09:14:59.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b337fe640 1 -- 172.21.15.111:0/737753641 learned_addr learned my addr 172.21.15.111:0/737753641 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b2affd640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b341043b0 0x7f2b34199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b33fff640 1 -- 172.21.15.111:0/737753641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b341043b0 msgr2=0x7f2b34199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b33fff640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b341043b0 0x7f2b34199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b33fff640 1 -- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b34101f90 msgr2=0x7f2b341995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b33fff640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b34101f90 0x7f2b341995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b33fff640 1 -- 172.21.15.111:0/737753641 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b341a49b0 con 0x7f2b34104d50 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b337fe640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b34101f90 0x7f2b341995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b33fff640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b34104d50 0x7f2b3419a070 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00ec50 tx=0x7f2b2c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b317fa640 1 -- 172.21.15.111:0/737753641 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c003b70 con 0x7f2b34104d50 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b317fa640 1 -- 172.21.15.111:0/737753641 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f2b2c003d10 con 0x7f2b34104d50 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b341a4ca0 con 0x7f2b34104d50 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b317fa640 1 -- 172.21.15.111:0/737753641 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c010690 con 0x7f2b34104d50 2024-09-16T09:14:59.137 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.132+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b341a5170 con 0x7f2b34104d50 2024-09-16T09:14:59.138 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.133+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b34102370 con 0x7f2b34104d50 2024-09-16T09:14:59.138 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.134+0000 7f2b317fa640 1 -- 172.21.15.111:0/737753641 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f2b2c003fc0 con 0x7f2b34104d50 2024-09-16T09:14:59.138 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.134+0000 7f2b317fa640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2b2007ae10 0x7f2b2007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:14:59.138 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.134+0000 7f2b337fe640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2b2007ae10 0x7f2b2007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:14:59.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.134+0000 7f2b317fa640 1 -- 172.21.15.111:0/737753641 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(737..737 src has 1..737) ==== 8675+0+0 (secure 0 0 0) 0x7f2b2c09ef10 con 0x7f2b34104d50 2024-09-16T09:14:59.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.135+0000 7f2b337fe640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2b2007ae10 0x7f2b2007d2d0 secure :-1 s=READY pgs=3917 cs=0 l=1 rev1=1 crypto rx=0x7f2b24011550 tx=0x7f2b24005f90 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:14:59.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.139+0000 7f2b317fa640 1 -- 172.21.15.111:0/737753641 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f2b2c069c80 con 0x7f2b34104d50 2024-09-16T09:14:59.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:58 smithi111 ceph-mon[40056]: osdmap e736: 8 total, 8 up, 8 in 2024-09-16T09:14:59.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:59.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:58 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:59.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:58 smithi111 ceph-mon[29803]: osdmap e736: 8 total, 8 up, 8 in 2024-09-16T09:14:59.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:59.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:58 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:59.318 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.317+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} v 0) -- 0x7f2b3406a6e0 con 0x7f2b34104d50 2024-09-16T09:14:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:58 smithi114 ceph-mon[33315]: osdmap e736: 8 total, 8 up, 8 in 2024-09-16T09:14:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-111"} : dispatch 2024-09-16T09:14:59.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:58 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4175387409' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi111-76716-111"}]': finished 2024-09-16T09:14:59.929 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:14:59.928+0000 7f2b317fa640 1 -- 172.21.15.111:0/737753641 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12}]=0 pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' created v738) ==== 176+0+0 (secure 0 0 0) 0x7f2b2c06eb30 con 0x7f2b34104d50 2024-09-16T09:15:00.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.039+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} v 0) -- 0x7f2b3402e5c0 con 0x7f2b34104d50 2024-09-16T09:15:00.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.040+0000 7f2b317fa640 1 -- 172.21.15.111:0/737753641 <== mon.2 v2:172.21.15.111:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12}]=0 pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' already exists v738) ==== 183+0+0 (secure 0 0 0) 0x7f2b2c09c930 con 0x7f2b34104d50 2024-09-16T09:15:00.041 INFO:tasks.workunit.client.0.smithi111.stderr:pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' already exists 2024-09-16T09:15:00.043 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2b2007ae10 msgr2=0x7f2b2007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:00.043 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2b2007ae10 0x7f2b2007d2d0 secure :-1 s=READY pgs=3917 cs=0 l=1 rev1=1 crypto rx=0x7f2b24011550 tx=0x7f2b24005f90 comp rx=0 tx=0).stop 2024-09-16T09:15:00.043 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b34104d50 msgr2=0x7f2b3419a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:00.043 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b34104d50 0x7f2b3419a070 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00ec50 tx=0x7f2b2c00c6a0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.043 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 shutdown_connections 2024-09-16T09:15:00.043 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2b2007ae10 0x7f2b2007d2d0 unknown :-1 s=CLOSED pgs=3917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.044 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b34104d50 0x7f2b3419a070 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.044 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b341043b0 0x7f2b34199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.044 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 --2- 172.21.15.111:0/737753641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b34101f90 0x7f2b341995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.044 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 >> 172.21.15.111:0/737753641 conn(0x7f2b340776a0 msgr2=0x7f2b34105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:15:00.044 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 shutdown_connections 2024-09-16T09:15:00.044 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.042+0000 7f2b39a6a640 1 -- 172.21.15.111:0/737753641 wait complete. 2024-09-16T09:15:00.052 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool set-quota aa242ffd-5c69-42f2-92b2-30d0602f301c max_objects 10 2024-09-16T09:15:00.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.133+0000 7f0f590b0640 1 -- 172.21.15.111:0/3411014788 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f54109070 msgr2=0x7f0f5410f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:00.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.133+0000 7f0f590b0640 1 --2- 172.21.15.111:0/3411014788 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f54109070 0x7f0f5410f870 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009900 tx=0x7f0f4c02a410 comp rx=0 tx=0).stop 2024-09-16T09:15:00.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.133+0000 7f0f590b0640 1 -- 172.21.15.111:0/3411014788 shutdown_connections 2024-09-16T09:15:00.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.133+0000 7f0f590b0640 1 --2- 172.21.15.111:0/3411014788 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f54109070 0x7f0f5410f870 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.133+0000 7f0f590b0640 1 --2- 172.21.15.111:0/3411014788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f541086f0 0x7f0f54108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.135 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.133+0000 7f0f590b0640 1 --2- 172.21.15.111:0/3411014788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f54102cf0 0x7f0f541030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.133+0000 7f0f590b0640 1 -- 172.21.15.111:0/3411014788 >> 172.21.15.111:0/3411014788 conn(0x7f0f540fe750 msgr2=0x7f0f54100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:15:00.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.133+0000 7f0f590b0640 1 -- 172.21.15.111:0/3411014788 shutdown_connections 2024-09-16T09:15:00.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.133+0000 7f0f590b0640 1 -- 172.21.15.111:0/3411014788 wait complete. 2024-09-16T09:15:00.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.134+0000 7f0f590b0640 1 Processor -- start 2024-09-16T09:15:00.136 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.134+0000 7f0f590b0640 1 -- start start 2024-09-16T09:15:00.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.134+0000 7f0f590b0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f54102cf0 0x7f0f54198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:00.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.134+0000 7f0f590b0640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f541086f0 0x7f0f54198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:00.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.134+0000 7f0f590b0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f54109070 0x7f0f5419e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:00.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.134+0000 7f0f590b0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0f541121c0 con 0x7f0f54109070 2024-09-16T09:15:00.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.134+0000 7f0f590b0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f54112040 con 0x7f0f54102cf0 2024-09-16T09:15:00.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.134+0000 7f0f590b0640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f0f54112340 con 0x7f0f541086f0 2024-09-16T09:15:00.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.134+0000 7f0f5259c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f541086f0 0x7f0f54198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:00.139 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5359e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f54109070 0x7f0f5419e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5259c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f541086f0 0x7f0f54198cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:51882/0 (socket says 172.21.15.111:51882) 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5259c640 1 -- 172.21.15.111:0/1308946484 learned_addr learned my addr 172.21.15.111:0/1308946484 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f52d9d640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f54102cf0 0x7f0f54198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5259c640 1 -- 172.21.15.111:0/1308946484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f54102cf0 msgr2=0x7f0f54198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5259c640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f54102cf0 0x7f0f54198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5259c640 1 -- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f54109070 msgr2=0x7f0f5419e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5259c640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f54109070 0x7f0f5419e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5259c640 1 -- 172.21.15.111:0/1308946484 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f5419e820 con 0x7f0f541086f0 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5359e640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f54109070 0x7f0f5419e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f5259c640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f541086f0 0x7f0f54198cc0 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00d770 tx=0x7f0f3c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f4bfff640 1 -- 172.21.15.111:0/1308946484 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c017070 con 0x7f0f541086f0 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f4bfff640 1 -- 172.21.15.111:0/1308946484 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f0f3c005470 con 0x7f0f541086f0 2024-09-16T09:15:00.140 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f5419eb10 con 0x7f0f541086f0 2024-09-16T09:15:00.141 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f4bfff640 1 -- 172.21.15.111:0/1308946484 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c004b40 con 0x7f0f541086f0 2024-09-16T09:15:00.141 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.135+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f541a51d0 con 0x7f0f541086f0 2024-09-16T09:15:00.141 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.136+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f541030d0 con 0x7f0f541086f0 2024-09-16T09:15:00.141 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.137+0000 7f0f4bfff640 1 -- 172.21.15.111:0/1308946484 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f0f3c004ce0 con 0x7f0f541086f0 2024-09-16T09:15:00.141 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.137+0000 7f0f4bfff640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f0f4007ae10 0x7f0f4007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:00.141 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.138+0000 7f0f52d9d640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f0f4007ae10 0x7f0f4007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:00.141 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.138+0000 7f0f4bfff640 1 -- 172.21.15.111:0/1308946484 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(738..738 src has 1..738) ==== 9050+0+0 (secure 0 0 0) 0x7f0f3c09e530 con 0x7f0f541086f0 2024-09-16T09:15:00.141 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.138+0000 7f0f52d9d640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f0f4007ae10 0x7f0f4007d2d0 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 crypto rx=0x7f0f440099a0 tx=0x7f0f44009500 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:15:00.143 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.142+0000 7f0f4bfff640 1 -- 172.21.15.111:0/1308946484 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f0f3c069130 con 0x7f0f541086f0 2024-09-16T09:15:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[29803]: pgmap v1140: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:15:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[29803]: osdmap e737: 8 total, 8 up, 8 in 2024-09-16T09:15:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:15:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/737753641' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[40056]: pgmap v1140: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:15:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[40056]: osdmap e737: 8 total, 8 up, 8 in 2024-09-16T09:15:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:15:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/737753641' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:00.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:14:59 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:00.317 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.316+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} v 0) -- 0x7f0f5410ae50 con 0x7f0f541086f0 2024-09-16T09:15:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:59 smithi114 ceph-mon[33315]: pgmap v1140: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:15:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:59 smithi114 ceph-mon[33315]: osdmap e737: 8 total, 8 up, 8 in 2024-09-16T09:15:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:59 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:15:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:59 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/737753641' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:00.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:14:59 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:00.936 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:00.935+0000 7f0f4bfff640 1 -- 172.21.15.111:0/1308946484 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v739) ==== 223+0+0 (secure 0 0 0) 0x7f0f3c06dfe0 con 0x7f0f541086f0 2024-09-16T09:15:01.050 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.049+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} v 0) -- 0x7f0f5410aa60 con 0x7f0f541086f0 2024-09-16T09:15:01.367 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12}]': finished 2024-09-16T09:15:01.367 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:00 smithi114 ceph-mon[33315]: osdmap e738: 8 total, 8 up, 8 in 2024-09-16T09:15:01.367 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/737753641' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:01.367 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:01.368 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:00 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1308946484' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:01.368 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:00 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12}]': finished 2024-09-16T09:15:01.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[29803]: osdmap e738: 8 total, 8 up, 8 in 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/737753641' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1308946484' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12}]': finished 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[40056]: osdmap e738: 8 total, 8 up, 8 in 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/737753641' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pg_num": 12} : dispatch 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1308946484' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:01.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:00 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:01.942 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.941+0000 7f0f4bfff640 1 -- 172.21.15.111:0/1308946484 <== mon.2 v2:172.21.15.111:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v740) ==== 223+0+0 (secure 0 0 0) 0x7f0f3c09bde0 con 0x7f0f541086f0 2024-09-16T09:15:01.942 INFO:tasks.workunit.client.0.smithi111.stderr:set-quota max_objects = 10 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f0f4007ae10 msgr2=0x7f0f4007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f0f4007ae10 0x7f0f4007d2d0 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 crypto rx=0x7f0f440099a0 tx=0x7f0f44009500 comp rx=0 tx=0).stop 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f541086f0 msgr2=0x7f0f54198cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f541086f0 0x7f0f54198cc0 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00d770 tx=0x7f0f3c00dc30 comp rx=0 tx=0).stop 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 shutdown_connections 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f0f4007ae10 0x7f0f4007d2d0 unknown :-1 s=CLOSED pgs=3918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f54109070 0x7f0f5419e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f541086f0 0x7f0f54198cc0 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 --2- 172.21.15.111:0/1308946484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f54102cf0 0x7f0f54198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.943+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 >> 172.21.15.111:0/1308946484 conn(0x7f0f540fe750 msgr2=0x7f0f54100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.944+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 shutdown_connections 2024-09-16T09:15:01.945 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:01.944+0000 7f0f590b0640 1 -- 172.21.15.111:0/1308946484 wait complete. 2024-09-16T09:15:01.954 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool application enable aa242ffd-5c69-42f2-92b2-30d0602f301c rados 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.035+0000 7f671566b640 1 -- 172.21.15.111:0/3005688559 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710102d00 msgr2=0x7f67101030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.035+0000 7f671566b640 1 --2- 172.21.15.111:0/3005688559 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710102d00 0x7f67101030e0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f6708009ca0 tx=0x7f670802a8c0 comp rx=0 tx=0).stop 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.035+0000 7f671566b640 1 -- 172.21.15.111:0/3005688559 shutdown_connections 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.035+0000 7f671566b640 1 --2- 172.21.15.111:0/3005688559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6710109080 0x7f671010f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.035+0000 7f671566b640 1 --2- 172.21.15.111:0/3005688559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6710108700 0x7f6710108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.035+0000 7f671566b640 1 --2- 172.21.15.111:0/3005688559 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710102d00 0x7f67101030e0 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.035+0000 7f671566b640 1 -- 172.21.15.111:0/3005688559 >> 172.21.15.111:0/3005688559 conn(0x7f67100fe740 msgr2=0x7f6710100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.035+0000 7f671566b640 1 -- 172.21.15.111:0/3005688559 shutdown_connections 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.035+0000 7f671566b640 1 -- 172.21.15.111:0/3005688559 wait complete. 2024-09-16T09:15:02.037 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f671566b640 1 Processor -- start 2024-09-16T09:15:02.038 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f671566b640 1 -- start start 2024-09-16T09:15:02.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f671566b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6710102d00 0x7f6710198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:02.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f671566b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6710108700 0x7f6710198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:02.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f671566b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710109080 0x7f671019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:02.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f671566b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f67101121d0 con 0x7f6710102d00 2024-09-16T09:15:02.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f671566b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6710112050 con 0x7f6710108700 2024-09-16T09:15:02.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f671566b640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f6710112350 con 0x7f6710109080 2024-09-16T09:15:02.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f670effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6710102d00 0x7f6710198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:02.040 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f670f7fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710109080 0x7f671019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f670f7fe640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710109080 0x7f671019dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:51908/0 (socket says 172.21.15.111:51908) 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f670f7fe640 1 -- 172.21.15.111:0/2221054981 learned_addr learned my addr 172.21.15.111:0/2221054981 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f670e7fc640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6710108700 0x7f6710198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f670f7fe640 1 -- 172.21.15.111:0/2221054981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6710108700 msgr2=0x7f6710198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.036+0000 7f670f7fe640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6710108700 0x7f6710198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f670f7fe640 1 -- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6710102d00 msgr2=0x7f6710198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f670f7fe640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6710102d00 0x7f6710198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f670f7fe640 1 -- 172.21.15.111:0/2221054981 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671019e7b0 con 0x7f6710109080 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f670effd640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6710102d00 0x7f6710198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f670f7fe640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710109080 0x7f671019dfe0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f670000c970 tx=0x7f670000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f6707fff640 1 -- 172.21.15.111:0/2221054981 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6700007bb0 con 0x7f6710109080 2024-09-16T09:15:02.041 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f6707fff640 1 -- 172.21.15.111:0/2221054981 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f6700007d50 con 0x7f6710109080 2024-09-16T09:15:02.042 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671019eaa0 con 0x7f6710109080 2024-09-16T09:15:02.042 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f6707fff640 1 -- 172.21.15.111:0/2221054981 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6700018660 con 0x7f6710109080 2024-09-16T09:15:02.042 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.037+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67101a5160 con 0x7f6710109080 2024-09-16T09:15:02.042 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.038+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67101030e0 con 0x7f6710109080 2024-09-16T09:15:02.042 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.038+0000 7f6707fff640 1 -- 172.21.15.111:0/2221054981 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f670000f450 con 0x7f6710109080 2024-09-16T09:15:02.042 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.039+0000 7f6707fff640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f66fc07ae10 0x7f66fc07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:02.042 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.039+0000 7f670effd640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f66fc07ae10 0x7f66fc07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:02.042 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.039+0000 7f6707fff640 1 -- 172.21.15.111:0/2221054981 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(740..740 src has 1..740) ==== 9050+0+0 (secure 0 0 0) 0x7f670009eb20 con 0x7f6710109080 2024-09-16T09:15:02.042 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.039+0000 7f670effd640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f66fc07ae10 0x7f66fc07d2d0 secure :-1 s=READY pgs=3919 cs=0 l=1 rev1=1 crypto rx=0x7f6708006010 tx=0x7f6708000f40 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:15:02.044 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.043+0000 7f6707fff640 1 -- 172.21.15.111:0/2221054981 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f6700014030 con 0x7f6710109080 2024-09-16T09:15:02.221 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.220+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} v 0) -- 0x7f671010ae60 con 0x7f6710109080 2024-09-16T09:15:02.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:01 smithi114 ceph-mon[33315]: pgmap v1143: 176 pgs: 12 unknown, 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:15:02.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:15:02.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:01 smithi114 ceph-mon[33315]: osdmap e739: 8 total, 8 up, 8 in 2024-09-16T09:15:02.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:01 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1308946484' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:02.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:01 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:02.345 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[29803]: pgmap v1143: 176 pgs: 12 unknown, 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:15:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:15:02.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[29803]: osdmap e739: 8 total, 8 up, 8 in 2024-09-16T09:15:02.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1308946484' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:02.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:02.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[40056]: pgmap v1143: 176 pgs: 12 unknown, 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-09-16T09:15:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:15:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[40056]: osdmap e739: 8 total, 8 up, 8 in 2024-09-16T09:15:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1308946484' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:15:02.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:02.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:15:02 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:15:02.947 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:02.946+0000 7f6707fff640 1 -- 172.21.15.111:0/2221054981 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"}]=0 enabled application 'rados' on pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' v741) ==== 213+0+0 (secure 0 0 0) 0x7f6700069720 con 0x7f6710109080 2024-09-16T09:15:03.053 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.052+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} v 0) -- 0x7f671010a8d0 con 0x7f6710109080 2024-09-16T09:15:03.327 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:15:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:15:03.328 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:15:03.328 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[29803]: osdmap e740: 8 total, 8 up, 8 in 2024-09-16T09:15:03.328 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2221054981' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:03.328 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:03.328 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:03.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:15:03.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[40056]: osdmap e740: 8 total, 8 up, 8 in 2024-09-16T09:15:03.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2221054981' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:03.328 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:03.329 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:15:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:02 smithi114 ceph-mon[33315]: osdmap e740: 8 total, 8 up, 8 in 2024-09-16T09:15:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:02 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2221054981' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:02 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:03.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:03.953 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.952+0000 7f6707fff640 1 -- 172.21.15.111:0/2221054981 <== mon.2 v2:172.21.15.111:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"}]=0 enabled application 'rados' on pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' v742) ==== 213+0+0 (secure 0 0 0) 0x7f670006e5d0 con 0x7f6710109080 2024-09-16T09:15:03.953 INFO:tasks.workunit.client.0.smithi111.stderr:enabled application 'rados' on pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' 2024-09-16T09:15:03.956 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f66fc07ae10 msgr2=0x7f66fc07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:03.956 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f66fc07ae10 0x7f66fc07d2d0 secure :-1 s=READY pgs=3919 cs=0 l=1 rev1=1 crypto rx=0x7f6708006010 tx=0x7f6708000f40 comp rx=0 tx=0).stop 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710109080 msgr2=0x7f671019dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710109080 0x7f671019dfe0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f670000c970 tx=0x7f670000ce30 comp rx=0 tx=0).stop 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 shutdown_connections 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f66fc07ae10 0x7f66fc07d2d0 unknown :-1 s=CLOSED pgs=3919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6710109080 0x7f671019dfe0 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6710108700 0x7f6710198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 --2- 172.21.15.111:0/2221054981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6710102d00 0x7f6710198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 >> 172.21.15.111:0/2221054981 conn(0x7f67100fe740 msgr2=0x7f6710100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 shutdown_connections 2024-09-16T09:15:03.957 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:03.955+0000 7f671566b640 1 -- 172.21.15.111:0/2221054981 wait complete. 2024-09-16T09:15:03.968 INFO:tasks.workunit.client.0.smithi111.stderr:++ seq 1 10 2024-09-16T09:15:03.980 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:03.980 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj1 /etc/passwd 2024-09-16T09:15:04.024 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:04.024 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj2 /etc/passwd 2024-09-16T09:15:04.064 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:04.064 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj3 /etc/passwd 2024-09-16T09:15:04.100 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:04.100 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj4 /etc/passwd 2024-09-16T09:15:04.136 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:04.136 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj5 /etc/passwd 2024-09-16T09:15:04.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:04.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj6 /etc/passwd 2024-09-16T09:15:04.208 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:04.208 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj7 /etc/passwd 2024-09-16T09:15:04.244 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:04.244 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj8 /etc/passwd 2024-09-16T09:15:04.280 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:04.280 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj9 /etc/passwd 2024-09-16T09:15:04.315 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:15:04.316 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put obj10 /etc/passwd 2024-09-16T09:15:04.346 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[29803]: pgmap v1146: 176 pgs: 176 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:04.346 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"}]': finished 2024-09-16T09:15:04.346 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[29803]: osdmap e741: 8 total, 8 up, 8 in 2024-09-16T09:15:04.346 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2221054981' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:04.347 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:04.347 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[40056]: pgmap v1146: 176 pgs: 176 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:04.347 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"}]': finished 2024-09-16T09:15:04.347 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[40056]: osdmap e741: 8 total, 8 up, 8 in 2024-09-16T09:15:04.347 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2221054981' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:04.347 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:03 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:04.351 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 30 2024-09-16T09:15:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:03 smithi114 ceph-mon[33315]: pgmap v1146: 176 pgs: 176 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"}]': finished 2024-09-16T09:15:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:03 smithi114 ceph-mon[33315]: osdmap e741: 8 total, 8 up, 8 in 2024-09-16T09:15:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:03 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2221054981' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:04.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:03 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"} : dispatch 2024-09-16T09:15:05.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:04 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"}]': finished 2024-09-16T09:15:05.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:04 smithi111 ceph-mon[29803]: osdmap e742: 8 total, 8 up, 8 in 2024-09-16T09:15:05.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:04 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"}]': finished 2024-09-16T09:15:05.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:04 smithi111 ceph-mon[40056]: osdmap e742: 8 total, 8 up, 8 in 2024-09-16T09:15:05.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:04 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "app": "rados"}]': finished 2024-09-16T09:15:05.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:04 smithi114 ceph-mon[33315]: osdmap e742: 8 total, 8 up, 8 in 2024-09-16T09:15:05.663 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:15:05 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=sqlstore.transactions t=2024-09-16T09:15:05.237983399Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-16T09:15:06.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:05 smithi114 ceph-mon[33315]: pgmap v1149: 176 pgs: 176 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:06.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:05 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:15:06.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:05 smithi111 ceph-mon[29803]: pgmap v1149: 176 pgs: 176 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:06.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:05 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:15:06.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:05 smithi111 ceph-mon[40056]: pgmap v1149: 176 pgs: 176 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:06.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:05 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:15:08.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:07 smithi114 ceph-mon[33315]: pgmap v1150: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 22 KiB/s wr, 3 op/s 2024-09-16T09:15:08.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:07 smithi111 ceph-mon[29803]: pgmap v1150: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 22 KiB/s wr, 3 op/s 2024-09-16T09:15:08.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:07 smithi111 ceph-mon[40056]: pgmap v1150: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 22 KiB/s wr, 3 op/s 2024-09-16T09:15:09.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:08 smithi114 ceph-mon[33315]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' is full (reached quota's max_objects: 10) 2024-09-16T09:15:09.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:08 smithi114 ceph-mon[33315]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:15:09.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:08 smithi114 ceph-mon[33315]: osdmap e743: 8 total, 8 up, 8 in 2024-09-16T09:15:09.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:08 smithi111 ceph-mon[29803]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' is full (reached quota's max_objects: 10) 2024-09-16T09:15:09.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:08 smithi111 ceph-mon[29803]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:15:09.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:08 smithi111 ceph-mon[29803]: osdmap e743: 8 total, 8 up, 8 in 2024-09-16T09:15:09.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:08 smithi111 ceph-mon[40056]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' is full (reached quota's max_objects: 10) 2024-09-16T09:15:09.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:08 smithi111 ceph-mon[40056]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:15:09.427 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:08 smithi111 ceph-mon[40056]: osdmap e743: 8 total, 8 up, 8 in 2024-09-16T09:15:10.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:10 smithi114 ceph-mon[33315]: pgmap v1152: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-16T09:15:10.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:10 smithi111 ceph-mon[29803]: pgmap v1152: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-16T09:15:10.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:10 smithi111 ceph-mon[40056]: pgmap v1152: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-16T09:15:12.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:12 smithi114 ceph-mon[33315]: pgmap v1153: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 656 B/s rd, 17 KiB/s wr, 1 op/s 2024-09-16T09:15:12.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:15:12 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:15:12.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:12 smithi111 ceph-mon[29803]: pgmap v1153: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 656 B/s rd, 17 KiB/s wr, 1 op/s 2024-09-16T09:15:12.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:12 smithi111 ceph-mon[40056]: pgmap v1153: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 656 B/s rd, 17 KiB/s wr, 1 op/s 2024-09-16T09:15:13.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:13 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:13.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:13 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:13.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:13 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:13.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:15:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:15:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:15:14.665 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:14 smithi114 ceph-mon[33315]: pgmap v1154: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-09-16T09:15:15.026 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:14 smithi111 ceph-mon[40056]: pgmap v1154: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-09-16T09:15:15.026 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:14 smithi111 ceph-mon[29803]: pgmap v1154: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-09-16T09:15:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:16 smithi111 ceph-mon[29803]: pgmap v1155: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:15:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:16 smithi111 ceph-mon[40056]: pgmap v1155: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:15:16.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:16 smithi114 ceph-mon[33315]: pgmap v1155: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:15:17.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:15:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:17 smithi114 ceph-mon[33315]: pgmap v1156: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:15:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:15:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:17 smithi111 ceph-mon[29803]: pgmap v1156: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:15:17.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:15:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:17.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:17 smithi111 ceph-mon[40056]: pgmap v1156: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:15:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:19 smithi114 ceph-mon[33315]: pgmap v1157: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-09-16T09:15:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:19 smithi111 ceph-mon[29803]: pgmap v1157: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-09-16T09:15:20.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:19 smithi111 ceph-mon[40056]: pgmap v1157: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-09-16T09:15:22.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:21 smithi114 ceph-mon[33315]: pgmap v1158: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:15:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:21 smithi111 ceph-mon[29803]: pgmap v1158: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:15:22.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:21 smithi111 ceph-mon[40056]: pgmap v1158: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:15:22.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:15:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:15:23.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:23.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:23.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:15:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:15:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:15:24.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:23 smithi114 ceph-mon[33315]: pgmap v1159: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:24.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:23 smithi111 ceph-mon[29803]: pgmap v1159: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:24.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:23 smithi111 ceph-mon[40056]: pgmap v1159: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:25 smithi114 ceph-mon[33315]: pgmap v1160: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:15:26.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:25 smithi111 ceph-mon[29803]: pgmap v1160: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:15:26.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:25 smithi111 ceph-mon[40056]: pgmap v1160: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:15:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:27 smithi114 ceph-mon[33315]: pgmap v1161: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:28.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:27 smithi111 ceph-mon[29803]: pgmap v1161: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:28.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:27 smithi111 ceph-mon[40056]: pgmap v1161: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:29 smithi114 ceph-mon[33315]: pgmap v1162: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:15:30.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:29 smithi111 ceph-mon[40056]: pgmap v1162: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:15:30.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:29 smithi111 ceph-mon[29803]: pgmap v1162: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:15:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:31 smithi114 ceph-mon[33315]: pgmap v1163: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:15:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:31 smithi111 ceph-mon[29803]: pgmap v1163: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:15:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:31 smithi111 ceph-mon[40056]: pgmap v1163: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:15:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:32.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:15:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:15:33.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:33.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:15:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:15:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:15:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:33 smithi114 ceph-mon[33315]: pgmap v1164: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:33 smithi111 ceph-mon[29803]: pgmap v1164: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:33 smithi111 ceph-mon[40056]: pgmap v1164: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:34.353 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=146601 2024-09-16T09:15:34.354 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool set-quota aa242ffd-5c69-42f2-92b2-30d0602f301c max_objects 100 2024-09-16T09:15:34.354 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put onemore /etc/passwd 2024-09-16T09:15:34.451 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.447+0000 7fd5f6eed640 1 -- 172.21.15.111:0/3712297427 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5f0109060 msgr2=0x7fd5f010f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:34.451 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.447+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/3712297427 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5f0109060 0x7fd5f010f860 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009ca0 tx=0x7fd5e802a8c0 comp rx=0 tx=0).stop 2024-09-16T09:15:34.451 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.447+0000 7fd5f6eed640 1 -- 172.21.15.111:0/3712297427 shutdown_connections 2024-09-16T09:15:34.451 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.447+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/3712297427 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5f0109060 0x7fd5f010f860 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:34.451 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.447+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/3712297427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5f01086e0 0x7fd5f0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:34.451 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.447+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/3712297427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5f0102ce0 0x7fd5f01030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.447+0000 7fd5f6eed640 1 -- 172.21.15.111:0/3712297427 >> 172.21.15.111:0/3712297427 conn(0x7fd5f00fe700 msgr2=0x7fd5f0100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.447+0000 7fd5f6eed640 1 -- 172.21.15.111:0/3712297427 shutdown_connections 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.447+0000 7fd5f6eed640 1 -- 172.21.15.111:0/3712297427 wait complete. 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f6eed640 1 Processor -- start 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f6eed640 1 -- start start 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f6eed640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5f0102ce0 0x7fd5f00754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f6eed640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5f01086e0 0x7fd5f00759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f6eed640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5f0109060 0x7fd5f0075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f6eed640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f01121b0 con 0x7fd5f0102ce0 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f6eed640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f0112030 con 0x7fd5f01086e0 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f6eed640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f0112330 con 0x7fd5f0109060 2024-09-16T09:15:34.452 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f4c89640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5f0102ce0 0x7fd5f00754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f4c89640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5f0102ce0 0x7fd5f00754a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49744/0 (socket says 172.21.15.111:49744) 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f4c89640 1 -- 172.21.15.111:0/1886555608 learned_addr learned my addr 172.21.15.111:0/1886555608 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f548a640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5f0109060 0x7fd5f0075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5effff640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5f01086e0 0x7fd5f00759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.448+0000 7fd5f4c89640 1 -- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5f0109060 msgr2=0x7fd5f0075f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5f4c89640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5f0109060 0x7fd5f0075f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5f4c89640 1 -- 172.21.15.111:0/1886555608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5f01086e0 msgr2=0x7fd5f00759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5f4c89640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5f01086e0 0x7fd5f00759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5f4c89640 1 -- 172.21.15.111:0/1886555608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f01a6b20 con 0x7fd5f0102ce0 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5effff640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5f01086e0 0x7fd5f00759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5f4c89640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5f0102ce0 0x7fd5f00754a0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400ea60 tx=0x7fd5e400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:15:34.453 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400cde0 con 0x7fd5f0102ce0 2024-09-16T09:15:34.454 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fd5e4004470 con 0x7fd5f0102ce0 2024-09-16T09:15:34.454 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f01a6e10 con 0x7fd5f0102ce0 2024-09-16T09:15:34.454 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4010430 con 0x7fd5f0102ce0 2024-09-16T09:15:34.454 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.449+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f01a72d0 con 0x7fd5f0102ce0 2024-09-16T09:15:34.454 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.450+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f01030c0 con 0x7fd5f0102ce0 2024-09-16T09:15:34.455 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.451+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7fd5e4003f60 con 0x7fd5f0102ce0 2024-09-16T09:15:34.455 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.451+0000 7fd5edffb640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fd5e007ae10 0x7fd5e007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:34.455 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.451+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(743..743 src has 1..743) ==== 9063+0+0 (secure 0 0 0) 0x7fd5e409eda0 con 0x7fd5f0102ce0 2024-09-16T09:15:34.455 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.451+0000 7fd5effff640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fd5e007ae10 0x7fd5e007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:34.455 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.451+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=744}) -- 0x7fd5e0086810 con 0x7fd5f0102ce0 2024-09-16T09:15:34.455 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.452+0000 7fd5effff640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fd5e007ae10 0x7fd5e007d2d0 secure :-1 s=READY pgs=3931 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0058c0 tx=0x7fd5dc005850 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:15:34.457 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.456+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fd5e4069990 con 0x7fd5f0102ce0 2024-09-16T09:15:34.636 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.635+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"} v 0) -- 0x7fd5f010ae40 con 0x7fd5f0102ce0 2024-09-16T09:15:34.841 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.840+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v744) ==== 225+0+0 (secure 0 0 0) 0x7fd5e406e840 con 0x7fd5f0102ce0 2024-09-16T09:15:34.843 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.843+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 8 ==== osd_map(744..744 src has 1..744) ==== 628+0+0 (secure 0 0 0) 0x7fd5e409c640 con 0x7fd5f0102ce0 2024-09-16T09:15:34.844 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.843+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=745}) -- 0x7fd5e0087770 con 0x7fd5f0102ce0 2024-09-16T09:15:34.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:34.948+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"} v 0) -- 0x7fd5f010a8b0 con 0x7fd5f0102ce0 2024-09-16T09:15:35.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:34 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"} : dispatch 2024-09-16T09:15:35.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:34 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"} : dispatch 2024-09-16T09:15:35.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:34 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"} : dispatch 2024-09-16T09:15:35.852 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.851+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v745) ==== 225+0+0 (secure 0 0 0) 0x7fd5e4003780 con 0x7fd5f0102ce0 2024-09-16T09:15:35.853 INFO:tasks.workunit.client.0.smithi111.stderr:set-quota max_objects = 100 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c 2024-09-16T09:15:35.854 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.853+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 <== mon.0 v2:172.21.15.111:3300/0 10 ==== osd_map(745..745 src has 1..745) ==== 628+0+0 (secure 0 0 0) 0x7fd5e406e3b0 con 0x7fd5f0102ce0 2024-09-16T09:15:35.854 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.853+0000 7fd5edffb640 1 -- 172.21.15.111:0/1886555608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=746}) -- 0x7fd5e0087980 con 0x7fd5f0102ce0 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.854+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fd5e007ae10 msgr2=0x7fd5e007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.854+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fd5e007ae10 0x7fd5e007d2d0 secure :-1 s=READY pgs=3931 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0058c0 tx=0x7fd5dc005850 comp rx=0 tx=0).stop 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.854+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5f0102ce0 msgr2=0x7fd5f00754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.854+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5f0102ce0 0x7fd5f00754a0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400ea60 tx=0x7fd5e400ef20 comp rx=0 tx=0).stop 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.855+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 shutdown_connections 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.855+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fd5e007ae10 0x7fd5e007d2d0 unknown :-1 s=CLOSED pgs=3931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.855+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5f0109060 0x7fd5f0075f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.855+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5f01086e0 0x7fd5f00759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.855+0000 7fd5f6eed640 1 --2- 172.21.15.111:0/1886555608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5f0102ce0 0x7fd5f00754a0 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.855+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 >> 172.21.15.111:0/1886555608 conn(0x7fd5f00fe700 msgr2=0x7fd5f0100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:15:35.856 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.855+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 shutdown_connections 2024-09-16T09:15:35.857 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:35.855+0000 7fd5f6eed640 1 -- 172.21.15.111:0/1886555608 wait complete. 2024-09-16T09:15:35.870 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 146601 2024-09-16T09:15:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:35 smithi114 ceph-mon[33315]: pgmap v1165: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:15:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"}]': finished 2024-09-16T09:15:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:35 smithi114 ceph-mon[33315]: osdmap e744: 8 total, 8 up, 8 in 2024-09-16T09:15:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:35 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"} : dispatch 2024-09-16T09:15:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:35 smithi111 ceph-mon[29803]: pgmap v1165: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:15:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"}]': finished 2024-09-16T09:15:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:35 smithi111 ceph-mon[29803]: osdmap e744: 8 total, 8 up, 8 in 2024-09-16T09:15:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:35 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"} : dispatch 2024-09-16T09:15:36.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:35 smithi111 ceph-mon[40056]: pgmap v1165: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:15:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"}]': finished 2024-09-16T09:15:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:35 smithi111 ceph-mon[40056]: osdmap e744: 8 total, 8 up, 8 in 2024-09-16T09:15:36.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:35 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"} : dispatch 2024-09-16T09:15:37.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:36 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"}]': finished 2024-09-16T09:15:37.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:36 smithi114 ceph-mon[33315]: osdmap e745: 8 total, 8 up, 8 in 2024-09-16T09:15:37.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:36 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"}]': finished 2024-09-16T09:15:37.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:36 smithi111 ceph-mon[29803]: osdmap e745: 8 total, 8 up, 8 in 2024-09-16T09:15:37.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:36 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/1886555608' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "100"}]': finished 2024-09-16T09:15:37.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:36 smithi111 ceph-mon[40056]: osdmap e745: 8 total, 8 up, 8 in 2024-09-16T09:15:38.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:37 smithi114 ceph-mon[33315]: pgmap v1168: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:37 smithi111 ceph-mon[29803]: pgmap v1168: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:38.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:37 smithi111 ceph-mon[40056]: pgmap v1168: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:38.408 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 0 -ne 0 ']' 2024-09-16T09:15:38.408 INFO:tasks.workunit.client.0.smithi111.stderr:+ true 2024-09-16T09:15:38.408 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put twomore /etc/passwd 2024-09-16T09:15:38.447 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool set-quota aa242ffd-5c69-42f2-92b2-30d0602f301c max_bytes 100 2024-09-16T09:15:38.533 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.530+0000 7fb2262cd640 1 -- 172.21.15.111:0/1610910571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2201076c0 msgr2=0x7fb220107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:38.533 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.530+0000 7fb2262cd640 1 --2- 172.21.15.111:0/1610910571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2201076c0 0x7fb220107b20 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7fb208009900 tx=0x7fb20802a410 comp rx=0 tx=0).stop 2024-09-16T09:15:38.533 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.530+0000 7fb2262cd640 1 -- 172.21.15.111:0/1610910571 shutdown_connections 2024-09-16T09:15:38.533 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.530+0000 7fb2262cd640 1 --2- 172.21.15.111:0/1610910571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb220108060 0x7fb22010e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:38.533 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.530+0000 7fb2262cd640 1 --2- 172.21.15.111:0/1610910571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2201076c0 0x7fb220107b20 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.530+0000 7fb2262cd640 1 --2- 172.21.15.111:0/1610910571 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb220102800 0x7fb220102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.530+0000 7fb2262cd640 1 -- 172.21.15.111:0/1610910571 >> 172.21.15.111:0/1610910571 conn(0x7fb220077fe0 msgr2=0x7fb2200783f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.530+0000 7fb2262cd640 1 -- 172.21.15.111:0/1610910571 shutdown_connections 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.530+0000 7fb2262cd640 1 -- 172.21.15.111:0/1610910571 wait complete. 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb2262cd640 1 Processor -- start 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb2262cd640 1 -- start start 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb2262cd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb220102800 0x7fb2201995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb2262cd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2201076c0 0x7fb220199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb2262cd640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb220108060 0x7fb22019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb2262cd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb220111090 con 0x7fb220102800 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb2262cd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb220110f10 con 0x7fb2201076c0 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb2262cd640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fb220111210 con 0x7fb220108060 2024-09-16T09:15:38.534 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb220102800 0x7fb2201995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb220102800 0x7fb2201995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49780/0 (socket says 172.21.15.111:49780) 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21ffff640 1 -- 172.21.15.111:0/4080990784 learned_addr learned my addr 172.21.15.111:0/4080990784 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb22486a640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb220108060 0x7fb22019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21ffff640 1 -- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb220108060 msgr2=0x7fb22019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21ffff640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb220108060 0x7fb22019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21ffff640 1 -- 172.21.15.111:0/4080990784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2201076c0 msgr2=0x7fb220199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21ffff640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2201076c0 0x7fb220199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21ffff640 1 -- 172.21.15.111:0/4080990784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2201a49c0 con 0x7fb220102800 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb22486a640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb220108060 0x7fb22019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21ffff640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb220102800 0x7fb2201995c0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fb21000ea60 tx=0x7fb21000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21dffb640 1 -- 172.21.15.111:0/4080990784 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21000cde0 con 0x7fb220102800 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21dffb640 1 -- 172.21.15.111:0/4080990784 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fb210004470 con 0x7fb220102800 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2201a4cb0 con 0x7fb220102800 2024-09-16T09:15:38.535 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.531+0000 7fb21dffb640 1 -- 172.21.15.111:0/4080990784 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210010430 con 0x7fb220102800 2024-09-16T09:15:38.536 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.532+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2201a5250 con 0x7fb220102800 2024-09-16T09:15:38.536 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.533+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb220102be0 con 0x7fb220102800 2024-09-16T09:15:38.537 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.533+0000 7fb21dffb640 1 -- 172.21.15.111:0/4080990784 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7fb210003f60 con 0x7fb220102800 2024-09-16T09:15:38.537 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.533+0000 7fb21dffb640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb20c07ae10 0x7fb20c07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:15:38.537 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.534+0000 7fb217fff640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb20c07ae10 0x7fb20c07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:15:38.537 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.534+0000 7fb21dffb640 1 -- 172.21.15.111:0/4080990784 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(746..746 src has 1..746) ==== 9063+0+0 (secure 0 0 0) 0x7fb21009ed70 con 0x7fb220102800 2024-09-16T09:15:38.537 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.534+0000 7fb217fff640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb20c07ae10 0x7fb20c07d2d0 secure :-1 s=READY pgs=3933 cs=0 l=1 rev1=1 crypto rx=0x7fb20802a930 tx=0x7fb208000bc0 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:15:38.539 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.538+0000 7fb21dffb640 1 -- 172.21.15.111:0/4080990784 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fb2100699e0 con 0x7fb220102800 2024-09-16T09:15:38.715 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:38.714+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"} v 0) -- 0x7fb220109db0 con 0x7fb220102800 2024-09-16T09:15:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:38 smithi114 ceph-mon[33315]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' no longer out of quota; removing NO_QUOTA flag 2024-09-16T09:15:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:38 smithi114 ceph-mon[33315]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:15:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:38 smithi114 ceph-mon[33315]: osdmap e746: 8 total, 8 up, 8 in 2024-09-16T09:15:39.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:38 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"} : dispatch 2024-09-16T09:15:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:38 smithi111 ceph-mon[29803]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' no longer out of quota; removing NO_QUOTA flag 2024-09-16T09:15:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:38 smithi111 ceph-mon[29803]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:15:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:38 smithi111 ceph-mon[29803]: osdmap e746: 8 total, 8 up, 8 in 2024-09-16T09:15:39.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:38 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"} : dispatch 2024-09-16T09:15:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:38 smithi111 ceph-mon[40056]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' no longer out of quota; removing NO_QUOTA flag 2024-09-16T09:15:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:38 smithi111 ceph-mon[40056]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:15:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:38 smithi111 ceph-mon[40056]: osdmap e746: 8 total, 8 up, 8 in 2024-09-16T09:15:39.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:38 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"} : dispatch 2024-09-16T09:15:39.400 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:39.399+0000 7fb21dffb640 1 -- 172.21.15.111:0/4080990784 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v747) ==== 221+0+0 (secure 0 0 0) 0x7fb21006e890 con 0x7fb220102800 2024-09-16T09:15:39.510 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:39.509+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"} v 0) -- 0x7fb220109820 con 0x7fb220102800 2024-09-16T09:15:40.402 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.401+0000 7fb21dffb640 1 -- 172.21.15.111:0/4080990784 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v748) ==== 221+0+0 (secure 0 0 0) 0x7fb21009c610 con 0x7fb220102800 2024-09-16T09:15:40.402 INFO:tasks.workunit.client.0.smithi111.stderr:set-quota max_bytes = 100 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c 2024-09-16T09:15:40.406 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.405+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb20c07ae10 msgr2=0x7fb20c07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:40.406 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.405+0000 7fb2262cd640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb20c07ae10 0x7fb20c07d2d0 secure :-1 s=READY pgs=3933 cs=0 l=1 rev1=1 crypto rx=0x7fb20802a930 tx=0x7fb208000bc0 comp rx=0 tx=0).stop 2024-09-16T09:15:40.406 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.405+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb220102800 msgr2=0x7fb2201995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:15:40.406 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.405+0000 7fb2262cd640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb220102800 0x7fb2201995c0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fb21000ea60 tx=0x7fb21000ef20 comp rx=0 tx=0).stop 2024-09-16T09:15:40.407 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.406+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 shutdown_connections 2024-09-16T09:15:40.407 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.406+0000 7fb2262cd640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fb20c07ae10 0x7fb20c07d2d0 unknown :-1 s=CLOSED pgs=3933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:40.407 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.406+0000 7fb2262cd640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb220108060 0x7fb22019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:40.407 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.406+0000 7fb2262cd640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2201076c0 0x7fb220199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:40.407 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.406+0000 7fb2262cd640 1 --2- 172.21.15.111:0/4080990784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb220102800 0x7fb2201995c0 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:15:40.408 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.406+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 >> 172.21.15.111:0/4080990784 conn(0x7fb220077fe0 msgr2=0x7fb2201003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:15:40.408 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.406+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 shutdown_connections 2024-09-16T09:15:40.408 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:15:40.406+0000 7fb2262cd640 1 -- 172.21.15.111:0/4080990784 wait complete. 2024-09-16T09:15:40.420 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 30 2024-09-16T09:15:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:40 smithi114 ceph-mon[33315]: pgmap v1170: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:15:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"}]': finished 2024-09-16T09:15:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:40 smithi114 ceph-mon[33315]: osdmap e747: 8 total, 8 up, 8 in 2024-09-16T09:15:40.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:40 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"} : dispatch 2024-09-16T09:15:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:40 smithi111 ceph-mon[29803]: pgmap v1170: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:15:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"}]': finished 2024-09-16T09:15:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:40 smithi111 ceph-mon[29803]: osdmap e747: 8 total, 8 up, 8 in 2024-09-16T09:15:40.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:40 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"} : dispatch 2024-09-16T09:15:40.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:40 smithi111 ceph-mon[40056]: pgmap v1170: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-16T09:15:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"}]': finished 2024-09-16T09:15:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:40 smithi111 ceph-mon[40056]: osdmap e747: 8 total, 8 up, 8 in 2024-09-16T09:15:40.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:40 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"} : dispatch 2024-09-16T09:15:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:41 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"}]': finished 2024-09-16T09:15:41.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:41 smithi114 ceph-mon[33315]: osdmap e748: 8 total, 8 up, 8 in 2024-09-16T09:15:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:41 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"}]': finished 2024-09-16T09:15:41.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:41 smithi111 ceph-mon[29803]: osdmap e748: 8 total, 8 up, 8 in 2024-09-16T09:15:41.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:41 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4080990784' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "100"}]': finished 2024-09-16T09:15:41.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:41 smithi111 ceph-mon[40056]: osdmap e748: 8 total, 8 up, 8 in 2024-09-16T09:15:42.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:42 smithi114 ceph-mon[33315]: pgmap v1173: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T09:15:42.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:15:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:15:42.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:42 smithi111 ceph-mon[29803]: pgmap v1173: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T09:15:42.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:42 smithi111 ceph-mon[40056]: pgmap v1173: 176 pgs: 176 active+clean; 708 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-16T09:15:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:43 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:43 smithi111 ceph-mon[29803]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' is full (reached quota's max_bytes: 100 B) 2024-09-16T09:15:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:43 smithi111 ceph-mon[29803]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:15:43.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:43 smithi111 ceph-mon[29803]: osdmap e749: 8 total, 8 up, 8 in 2024-09-16T09:15:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:43 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:43 smithi111 ceph-mon[40056]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' is full (reached quota's max_bytes: 100 B) 2024-09-16T09:15:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:43 smithi111 ceph-mon[40056]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:15:43.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:43 smithi111 ceph-mon[40056]: osdmap e749: 8 total, 8 up, 8 in 2024-09-16T09:15:43.677 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:15:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:15:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:15:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:43 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:43 smithi114 ceph-mon[33315]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' is full (reached quota's max_bytes: 100 B) 2024-09-16T09:15:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:43 smithi114 ceph-mon[33315]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:15:43.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:43 smithi114 ceph-mon[33315]: osdmap e749: 8 total, 8 up, 8 in 2024-09-16T09:15:44.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:44 smithi111 ceph-mon[29803]: pgmap v1174: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-16T09:15:44.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:44 smithi111 ceph-mon[40056]: pgmap v1174: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-16T09:15:44.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:44 smithi114 ceph-mon[33315]: pgmap v1174: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-16T09:15:46.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:46 smithi114 ceph-mon[33315]: pgmap v1176: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-16T09:15:46.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:46 smithi111 ceph-mon[29803]: pgmap v1176: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-16T09:15:46.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:46 smithi111 ceph-mon[40056]: pgmap v1176: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-16T09:15:47.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:47 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:47.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:47 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:47.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:47 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:15:48.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:48 smithi114 ceph-mon[33315]: pgmap v1177: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.5 KiB/s wr, 1 op/s 2024-09-16T09:15:48.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:48 smithi111 ceph-mon[29803]: pgmap v1177: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.5 KiB/s wr, 1 op/s 2024-09-16T09:15:48.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:48 smithi111 ceph-mon[40056]: pgmap v1177: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.5 KiB/s wr, 1 op/s 2024-09-16T09:15:50.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:50 smithi114 ceph-mon[33315]: pgmap v1178: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.1 KiB/s wr, 1 op/s 2024-09-16T09:15:50.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:50 smithi111 ceph-mon[29803]: pgmap v1178: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.1 KiB/s wr, 1 op/s 2024-09-16T09:15:50.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:50 smithi111 ceph-mon[40056]: pgmap v1178: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.1 KiB/s wr, 1 op/s 2024-09-16T09:15:52.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:52 smithi114 ceph-mon[33315]: pgmap v1179: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.6 KiB/s wr, 1 op/s 2024-09-16T09:15:52.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:15:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:15:52.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:52 smithi111 ceph-mon[29803]: pgmap v1179: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.6 KiB/s wr, 1 op/s 2024-09-16T09:15:52.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:52 smithi111 ceph-mon[40056]: pgmap v1179: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.6 KiB/s wr, 1 op/s 2024-09-16T09:15:53.446 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:15:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:15:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:15:53.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:53.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:53.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:15:54.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:54 smithi114 ceph-mon[33315]: pgmap v1180: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:15:54.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:54 smithi111 ceph-mon[29803]: pgmap v1180: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:15:54.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:54 smithi111 ceph-mon[40056]: pgmap v1180: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:15:55.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:55 smithi114 ceph-mon[33315]: pgmap v1181: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-09-16T09:15:55.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:55 smithi111 ceph-mon[29803]: pgmap v1181: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-09-16T09:15:55.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:55 smithi111 ceph-mon[40056]: pgmap v1181: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-09-16T09:15:57.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:56 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:15:57.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:56 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:15:57.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:56 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:15:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:57 smithi114 ceph-mon[33315]: pgmap v1182: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:57 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:15:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:57 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:15:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:57 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:15:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:57 smithi111 ceph-mon[40056]: pgmap v1182: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:57 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:15:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:57 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:15:58.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:57 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:15:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:57 smithi111 ceph-mon[29803]: pgmap v1182: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:15:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:57 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:15:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:57 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:15:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:57 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:16:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:15:59 smithi114 ceph-mon[33315]: pgmap v1183: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:15:59 smithi111 ceph-mon[29803]: pgmap v1183: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:00.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:15:59 smithi111 ceph-mon[40056]: pgmap v1183: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:01 smithi114 ceph-mon[33315]: pgmap v1184: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:01 smithi111 ceph-mon[29803]: pgmap v1184: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:02.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:01 smithi111 ceph-mon[40056]: pgmap v1184: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:02.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:02.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:16:02 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:16:03.077 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:03.078 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:03.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:03.426 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:16:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:16:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:16:04.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:03 smithi114 ceph-mon[33315]: pgmap v1185: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:04.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:03 smithi111 ceph-mon[29803]: pgmap v1185: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:04.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:03 smithi111 ceph-mon[40056]: pgmap v1185: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:06.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:05 smithi114 ceph-mon[33315]: pgmap v1186: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:05 smithi111 ceph-mon[29803]: pgmap v1186: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:06.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:05 smithi111 ceph-mon[40056]: pgmap v1186: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:08.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:07 smithi114 ceph-mon[33315]: pgmap v1187: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:07 smithi111 ceph-mon[29803]: pgmap v1187: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:08.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:07 smithi111 ceph-mon[40056]: pgmap v1187: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:10.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:09 smithi114 ceph-mon[33315]: pgmap v1188: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:10.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:09 smithi111 ceph-mon[29803]: pgmap v1188: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:10.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:09 smithi111 ceph-mon[40056]: pgmap v1188: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:10.422 INFO:tasks.workunit.client.0.smithi111.stderr:+ pid=146769 2024-09-16T09:16:10.422 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool set-quota aa242ffd-5c69-42f2-92b2-30d0602f301c max_bytes 0 2024-09-16T09:16:10.422 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put two /etc/passwd 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.516+0000 7f2f6f840640 1 -- 172.21.15.111:0/2554600585 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2f681043b0 msgr2=0x7f2f68104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.516+0000 7f2f6f840640 1 --2- 172.21.15.111:0/2554600585 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2f681043b0 0x7f2f68104810 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009900 tx=0x7f2f5c02a410 comp rx=0 tx=0).stop 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.516+0000 7f2f6f840640 1 -- 172.21.15.111:0/2554600585 shutdown_connections 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.516+0000 7f2f6f840640 1 --2- 172.21.15.111:0/2554600585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f68104d50 0x7f2f6810e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.516+0000 7f2f6f840640 1 --2- 172.21.15.111:0/2554600585 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2f681043b0 0x7f2f68104810 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.516+0000 7f2f6f840640 1 --2- 172.21.15.111:0/2554600585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f68101f90 0x7f2f68102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.516+0000 7f2f6f840640 1 -- 172.21.15.111:0/2554600585 >> 172.21.15.111:0/2554600585 conn(0x7f2f680776a0 msgr2=0x7f2f68077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.516+0000 7f2f6f840640 1 -- 172.21.15.111:0/2554600585 shutdown_connections 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.516+0000 7f2f6f840640 1 -- 172.21.15.111:0/2554600585 wait complete. 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6f840640 1 Processor -- start 2024-09-16T09:16:10.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6f840640 1 -- start start 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6f840640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f68101f90 0x7f2f68199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6f840640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f681043b0 0x7f2f68199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6f840640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2f68104d50 0x7f2f681a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6f840640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2f681110a0 con 0x7f2f68101f90 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6f840640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f68110f20 con 0x7f2f681043b0 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6f840640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f2f68111220 con 0x7f2f68104d50 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6d5dc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f68101f90 0x7f2f68199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6d5dc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f68101f90 0x7f2f68199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50036/0 (socket says 172.21.15.111:50036) 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6dddd640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2f68104d50 0x7f2f681a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6d5dc640 1 -- 172.21.15.111:0/3382100850 learned_addr learned my addr 172.21.15.111:0/3382100850 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6cddb640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f681043b0 0x7f2f68199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6d5dc640 1 -- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2f68104d50 msgr2=0x7f2f681a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:10.521 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6d5dc640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2f68104d50 0x7f2f681a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6d5dc640 1 -- 172.21.15.111:0/3382100850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f681043b0 msgr2=0x7f2f68199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6d5dc640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f681043b0 0x7f2f68199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6d5dc640 1 -- 172.21.15.111:0/3382100850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f681a4af0 con 0x7f2f68101f90 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6dddd640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2f68104d50 0x7f2f681a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.517+0000 7f2f6cddb640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f681043b0 0x7f2f68199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.518+0000 7f2f6d5dc640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f68101f90 0x7f2f68199530 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009d50 tx=0x7f2f5800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.518+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58003b70 con 0x7f2f68101f90 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.518+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f2f58003d10 con 0x7f2f68101f90 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.518+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f681a4cc0 con 0x7f2f68101f90 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.518+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f580106c0 con 0x7f2f68101f90 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.518+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f681a5180 con 0x7f2f68101f90 2024-09-16T09:16:10.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.519+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f68102370 con 0x7f2f68101f90 2024-09-16T09:16:10.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.519+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f2f58004020 con 0x7f2f68101f90 2024-09-16T09:16:10.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.520+0000 7f2f667fc640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2f5007ae10 0x7f2f5007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:10.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.520+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(749..749 src has 1..749) ==== 9063+0+0 (secure 0 0 0) 0x7f2f5809f1f0 con 0x7f2f68101f90 2024-09-16T09:16:10.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.520+0000 7f2f6cddb640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2f5007ae10 0x7f2f5007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:10.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.520+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=750}) -- 0x7f2f50086890 con 0x7f2f68101f90 2024-09-16T09:16:10.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.521+0000 7f2f6cddb640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2f5007ae10 0x7f2f5007d2d0 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0057c0 tx=0x7f2f5c000bc0 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:10.525 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.525+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f2f58069de0 con 0x7f2f68101f90 2024-09-16T09:16:10.705 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.702+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} v 0) -- 0x7f2f6802e5c0 con 0x7f2f68101f90 2024-09-16T09:16:10.842 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.841+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v750) ==== 217+0+0 (secure 0 0 0) 0x7f2f5806ec90 con 0x7f2f68101f90 2024-09-16T09:16:10.843 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.842+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 8 ==== osd_map(750..750 src has 1..750) ==== 628+0+0 (secure 0 0 0) 0x7f2f580d6190 con 0x7f2f68101f90 2024-09-16T09:16:10.843 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.842+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=751}) -- 0x7f2f500877f0 con 0x7f2f68101f90 2024-09-16T09:16:10.951 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:10.950+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} v 0) -- 0x7f2f6802e810 con 0x7f2f68101f90 2024-09-16T09:16:11.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:10 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:16:11.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:10 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:16:11.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:10 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:16:11.844 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.843+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v751) ==== 217+0+0 (secure 0 0 0) 0x7f2f5809ca90 con 0x7f2f68101f90 2024-09-16T09:16:11.844 INFO:tasks.workunit.client.0.smithi111.stderr:set-quota max_bytes = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c 2024-09-16T09:16:11.845 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.844+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 <== mon.0 v2:172.21.15.111:3300/0 10 ==== osd_map(751..751 src has 1..751) ==== 628+0+0 (secure 0 0 0) 0x7f2f5806c0b0 con 0x7f2f68101f90 2024-09-16T09:16:11.845 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.844+0000 7f2f667fc640 1 -- 172.21.15.111:0/3382100850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=752}) -- 0x7f2f50087a00 con 0x7f2f68101f90 2024-09-16T09:16:11.849 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2f5007ae10 msgr2=0x7f2f5007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:11.849 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2f5007ae10 0x7f2f5007d2d0 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0057c0 tx=0x7f2f5c000bc0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.849 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f68101f90 msgr2=0x7f2f68199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:11.849 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f68101f90 0x7f2f68199530 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009d50 tx=0x7f2f5800c6a0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.849 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 shutdown_connections 2024-09-16T09:16:11.849 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f2f5007ae10 0x7f2f5007d2d0 unknown :-1 s=CLOSED pgs=3935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.849 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2f68104d50 0x7f2f681a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.850 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f681043b0 0x7f2f68199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.850 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 --2- 172.21.15.111:0/3382100850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f68101f90 0x7f2f68199530 secure :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009d50 tx=0x7f2f5800c6a0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.850 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 >> 172.21.15.111:0/3382100850 conn(0x7f2f680776a0 msgr2=0x7f2f68105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:11.850 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 shutdown_connections 2024-09-16T09:16:11.850 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.848+0000 7f2f6f840640 1 -- 172.21.15.111:0/3382100850 wait complete. 2024-09-16T09:16:11.862 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool set-quota aa242ffd-5c69-42f2-92b2-30d0602f301c max_objects 0 2024-09-16T09:16:11.948 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 -- 172.21.15.111:0/686887759 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 msgr2=0x7f6f201030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:11.948 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 --2- 172.21.15.111:0/686887759 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 0x7f6f201030e0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009860 tx=0x7f6f1402a340 comp rx=0 tx=0).stop 2024-09-16T09:16:11.948 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 -- 172.21.15.111:0/686887759 shutdown_connections 2024-09-16T09:16:11.948 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 --2- 172.21.15.111:0/686887759 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f20109080 0x7f6f2010f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 --2- 172.21.15.111:0/686887759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f20108700 0x7f6f20108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 --2- 172.21.15.111:0/686887759 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 0x7f6f201030e0 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 -- 172.21.15.111:0/686887759 >> 172.21.15.111:0/686887759 conn(0x7f6f200fe740 msgr2=0x7f6f20100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:11.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 -- 172.21.15.111:0/686887759 shutdown_connections 2024-09-16T09:16:11.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 -- 172.21.15.111:0/686887759 wait complete. 2024-09-16T09:16:11.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 Processor -- start 2024-09-16T09:16:11.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.945+0000 7f6f27260640 1 -- start start 2024-09-16T09:16:11.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f27260640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 0x7f6f200754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:11.949 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f27260640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f20108700 0x7f6f200759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f27260640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f20109080 0x7f6f20075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f27260640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20112210 con 0x7f6f20108700 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f27260640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20112090 con 0x7f6f20109080 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f27260640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f6f20112390 con 0x7f6f20102d00 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f24ffc640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 0x7f6f200754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f1ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f20108700 0x7f6f200759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f24ffc640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 0x7f6f200754a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:55132/0 (socket says 172.21.15.111:55132) 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f24ffc640 1 -- 172.21.15.111:0/3193507184 learned_addr learned my addr 172.21.15.111:0/3193507184 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f24ffc640 1 -- 172.21.15.111:0/3193507184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f20109080 msgr2=0x7f6f20075f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:11.950 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f24ffc640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f20109080 0x7f6f20075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.951 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f24ffc640 1 -- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f20108700 msgr2=0x7f6f200759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:11.951 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f24ffc640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f20108700 0x7f6f200759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:11.951 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f24ffc640 1 -- 172.21.15.111:0/3193507184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f201a6cb0 con 0x7f6f20102d00 2024-09-16T09:16:11.951 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f1ffff640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f20108700 0x7f6f200759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:16:11.951 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f24ffc640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 0x7f6f200754a0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f6f14035040 tx=0x7f6f14033d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:11.951 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1403f020 con 0x7f6f20102d00 2024-09-16T09:16:11.951 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f6f140306b0 con 0x7f6f20102d00 2024-09-16T09:16:11.951 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f201a6f40 con 0x7f6f20102d00 2024-09-16T09:16:11.952 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14037040 con 0x7f6f20102d00 2024-09-16T09:16:11.952 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.946+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f201a73a0 con 0x7f6f20102d00 2024-09-16T09:16:11.953 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.947+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f201030e0 con 0x7f6f20102d00 2024-09-16T09:16:11.953 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.948+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f6f14042020 con 0x7f6f20102d00 2024-09-16T09:16:11.953 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.948+0000 7f6f1dffb640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f6f1007ae10 0x7f6f1007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:11.953 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.948+0000 7f6f1ffff640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f6f1007ae10 0x7f6f1007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:11.953 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.949+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(751..751 src has 1..751) ==== 9063+0+0 (secure 0 0 0) 0x7f6f140be850 con 0x7f6f20102d00 2024-09-16T09:16:11.953 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.949+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=752}) -- 0x7f6f10086890 con 0x7f6f20102d00 2024-09-16T09:16:11.953 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.949+0000 7f6f1ffff640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f6f1007ae10 0x7f6f1007d2d0 secure :-1 s=READY pgs=3936 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0096c0 tx=0x7f6f0c009260 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:11.954 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:11.953+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f6f14049020 con 0x7f6f20102d00 2024-09-16T09:16:12.129 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:12.128+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} v 0) -- 0x7f6f201a7690 con 0x7f6f20102d00 2024-09-16T09:16:12.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:11 smithi114 ceph-mon[33315]: pgmap v1189: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:12.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:16:12.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:11 smithi114 ceph-mon[33315]: osdmap e750: 8 total, 8 up, 8 in 2024-09-16T09:16:12.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:11 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:16:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:11 smithi111 ceph-mon[29803]: pgmap v1189: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:16:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:11 smithi111 ceph-mon[29803]: osdmap e750: 8 total, 8 up, 8 in 2024-09-16T09:16:12.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:11 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:16:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:11 smithi111 ceph-mon[40056]: pgmap v1189: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:16:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:11 smithi111 ceph-mon[40056]: osdmap e750: 8 total, 8 up, 8 in 2024-09-16T09:16:12.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:11 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:16:12.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:16:12 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:16:12.864 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:12.863+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 7 ==== osd_map(752..752 src has 1..752) ==== 628+0+0 (secure 0 0 0) 0x7f6f140bc0f0 con 0x7f6f20102d00 2024-09-16T09:16:12.864 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:12.863+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=753}) -- 0x7f6f10087400 con 0x7f6f20102d00 2024-09-16T09:16:12.869 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:12.868+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v752) ==== 221+0+0 (secure 0 0 0) 0x7f6f14089440 con 0x7f6f20102d00 2024-09-16T09:16:12.979 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:12.978+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} v 0) -- 0x7f6f201098f0 con 0x7f6f20102d00 2024-09-16T09:16:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:16:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:12 smithi114 ceph-mon[33315]: osdmap e751: 8 total, 8 up, 8 in 2024-09-16T09:16:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:12 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3193507184' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:12 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:13.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:12 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:16:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[29803]: osdmap e751: 8 total, 8 up, 8 in 2024-09-16T09:16:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3193507184' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:13.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3382100850' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:16:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[40056]: osdmap e751: 8 total, 8 up, 8 in 2024-09-16T09:16:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3193507184' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:13.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:12 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:13.405 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.404+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 9 ==== osd_map(753..753 src has 1..753) ==== 628+0+0 (secure 0 0 0) 0x7f6f1408de60 con 0x7f6f20102d00 2024-09-16T09:16:13.406 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.404+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=754}) -- 0x7f6f10087830 con 0x7f6f20102d00 2024-09-16T09:16:13.411 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.410+0000 7f6f1dffb640 1 -- 172.21.15.111:0/3193507184 <== mon.2 v2:172.21.15.111:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v753) ==== 221+0+0 (secure 0 0 0) 0x7f6f1408e2f0 con 0x7f6f20102d00 2024-09-16T09:16:13.411 INFO:tasks.workunit.client.0.smithi111.stderr:set-quota max_objects = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f6f1007ae10 msgr2=0x7f6f1007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f6f1007ae10 0x7f6f1007d2d0 secure :-1 s=READY pgs=3936 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0096c0 tx=0x7f6f0c009260 comp rx=0 tx=0).stop 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 msgr2=0x7f6f200754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 0x7f6f200754a0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f6f14035040 tx=0x7f6f14033d00 comp rx=0 tx=0).stop 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 shutdown_connections 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f6f1007ae10 0x7f6f1007d2d0 unknown :-1 s=CLOSED pgs=3936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f20109080 0x7f6f20075f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f20108700 0x7f6f200759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 --2- 172.21.15.111:0/3193507184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f20102d00 0x7f6f200754a0 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.412+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 >> 172.21.15.111:0/3193507184 conn(0x7f6f200fe740 msgr2=0x7f6f20101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.413+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 shutdown_connections 2024-09-16T09:16:13.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.413+0000 7f6f27260640 1 -- 172.21.15.111:0/3193507184 wait complete. 2024-09-16T09:16:13.423 INFO:tasks.workunit.client.0.smithi111.stderr:+ wait 146769 2024-09-16T09:16:13.424 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 0 -ne 0 ']' 2024-09-16T09:16:13.424 INFO:tasks.workunit.client.0.smithi111.stderr:+ true 2024-09-16T09:16:13.424 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put three /etc/passwd 2024-09-16T09:16:13.461 INFO:tasks.workunit.client.0.smithi111.stderr:++ uuidgen 2024-09-16T09:16:13.462 INFO:tasks.workunit.client.0.smithi111.stderr:+ pp=4b166c41-77f3-458e-9e25-c5fe356364a4 2024-09-16T09:16:13.462 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool create 4b166c41-77f3-458e-9e25-c5fe356364a4 12 2024-09-16T09:16:13.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.543+0000 7fae74589640 1 -- 172.21.15.111:0/1838332571 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c107f70 msgr2=0x7fae6c10e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:13.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.543+0000 7fae74589640 1 --2- 172.21.15.111:0/1838332571 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c107f70 0x7fae6c10e770 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7fae64009ca0 tx=0x7fae6402a8c0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.543+0000 7fae74589640 1 -- 172.21.15.111:0/1838332571 shutdown_connections 2024-09-16T09:16:13.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.543+0000 7fae74589640 1 --2- 172.21.15.111:0/1838332571 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c107f70 0x7fae6c10e770 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.543+0000 7fae74589640 1 --2- 172.21.15.111:0/1838332571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6c1075f0 0x7fae6c107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.543+0000 7fae74589640 1 --2- 172.21.15.111:0/1838332571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae6c101bf0 0x7fae6c101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.543+0000 7fae74589640 1 -- 172.21.15.111:0/1838332571 >> 172.21.15.111:0/1838332571 conn(0x7fae6c0fd630 msgr2=0x7fae6c0ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.543+0000 7fae74589640 1 -- 172.21.15.111:0/1838332571 shutdown_connections 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.543+0000 7fae74589640 1 -- 172.21.15.111:0/1838332571 wait complete. 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae74589640 1 Processor -- start 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae74589640 1 -- start start 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae74589640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c101bf0 0x7fae6c197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae74589640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae6c1075f0 0x7fae6c197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae74589640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6c107f70 0x7fae6c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae74589640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c1110c0 con 0x7fae6c1075f0 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae74589640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c110f40 con 0x7fae6c107f70 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae74589640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7fae6c111240 con 0x7fae6c101bf0 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae72325640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c101bf0 0x7fae6c197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:13.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae71b24640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae6c1075f0 0x7fae6c197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae72325640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c101bf0 0x7fae6c197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:55174/0 (socket says 172.21.15.111:55174) 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.544+0000 7fae72325640 1 -- 172.21.15.111:0/2219648310 learned_addr learned my addr 172.21.15.111:0/2219648310 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae72b26640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6c107f70 0x7fae6c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae72325640 1 -- 172.21.15.111:0/2219648310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6c107f70 msgr2=0x7fae6c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae72325640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6c107f70 0x7fae6c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae72325640 1 -- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae6c1075f0 msgr2=0x7fae6c197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae72325640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae6c1075f0 0x7fae6c197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae72325640 1 -- 172.21.15.111:0/2219648310 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6c19d6c0 con 0x7fae6c101bf0 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae71b24640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae6c1075f0 0x7fae6c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae72325640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c101bf0 0x7fae6c197620 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7fae6800ea60 tx=0x7fae6800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae637fe640 1 -- 172.21.15.111:0/2219648310 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6800cde0 con 0x7fae6c101bf0 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae637fe640 1 -- 172.21.15.111:0/2219648310 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7fae68004470 con 0x7fae6c101bf0 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6c19d9b0 con 0x7fae6c101bf0 2024-09-16T09:16:13.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae637fe640 1 -- 172.21.15.111:0/2219648310 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68010430 con 0x7fae6c101bf0 2024-09-16T09:16:13.550 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.545+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae6c1a4070 con 0x7fae6c101bf0 2024-09-16T09:16:13.550 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.546+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae6c101fd0 con 0x7fae6c101bf0 2024-09-16T09:16:13.550 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.547+0000 7fae637fe640 1 -- 172.21.15.111:0/2219648310 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7fae68003f60 con 0x7fae6c101bf0 2024-09-16T09:16:13.550 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.547+0000 7fae637fe640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fae5807ae10 0x7fae5807d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:13.550 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.547+0000 7fae71b24640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fae5807ae10 0x7fae5807d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:13.551 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.547+0000 7fae637fe640 1 -- 172.21.15.111:0/2219648310 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(753..753 src has 1..753) ==== 9063+0+0 (secure 0 0 0) 0x7fae6809ee60 con 0x7fae6c101bf0 2024-09-16T09:16:13.551 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.547+0000 7fae71b24640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fae5807ae10 0x7fae5807d2d0 secure :-1 s=READY pgs=3938 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0fe400 tx=0x7fae5c00b410 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:13.552 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.551+0000 7fae637fe640 1 -- 172.21.15.111:0/2219648310 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7fae68069a50 con 0x7fae6c101bf0 2024-09-16T09:16:13.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:16:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:16:13.728 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:13.727+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} v 0) -- 0x7fae6c109d50 con 0x7fae6c101bf0 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: pgmap v1192: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: osdmap e752: 8 total, 8 up, 8 in 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/3193507184' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' no longer out of quota; removing NO_QUOTA flag 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: osdmap e753: 8 total, 8 up, 8 in 2024-09-16T09:16:14.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2219648310' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:14.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:13 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:14.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: pgmap v1192: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:14.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:16:14.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: osdmap e752: 8 total, 8 up, 8 in 2024-09-16T09:16:14.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/3193507184' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:14.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:14.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' no longer out of quota; removing NO_QUOTA flag 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: osdmap e753: 8 total, 8 up, 8 in 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2219648310' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: pgmap v1192: 176 pgs: 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: osdmap e752: 8 total, 8 up, 8 in 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/3193507184' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' no longer out of quota; removing NO_QUOTA flag 2024-09-16T09:16:14.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:16:14.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:16:14.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: osdmap e753: 8 total, 8 up, 8 in 2024-09-16T09:16:14.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2219648310' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:14.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:13 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:14.409 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.408+0000 7fae637fe640 1 -- 172.21.15.111:0/2219648310 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12}]=0 pool '4b166c41-77f3-458e-9e25-c5fe356364a4' created v754) ==== 176+0+0 (secure 0 0 0) 0x7fae6806e900 con 0x7fae6c101bf0 2024-09-16T09:16:14.519 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.518+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} v 0) -- 0x7fae6c1097c0 con 0x7fae6c101bf0 2024-09-16T09:16:14.520 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.519+0000 7fae637fe640 1 -- 172.21.15.111:0/2219648310 <== mon.2 v2:172.21.15.111:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12}]=0 pool '4b166c41-77f3-458e-9e25-c5fe356364a4' already exists v754) ==== 183+0+0 (secure 0 0 0) 0x7fae68004250 con 0x7fae6c101bf0 2024-09-16T09:16:14.520 INFO:tasks.workunit.client.0.smithi111.stderr:pool '4b166c41-77f3-458e-9e25-c5fe356364a4' already exists 2024-09-16T09:16:14.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fae5807ae10 msgr2=0x7fae5807d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:14.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fae5807ae10 0x7fae5807d2d0 secure :-1 s=READY pgs=3938 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0fe400 tx=0x7fae5c00b410 comp rx=0 tx=0).stop 2024-09-16T09:16:14.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c101bf0 msgr2=0x7fae6c197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:14.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c101bf0 0x7fae6c197620 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7fae6800ea60 tx=0x7fae6800ef20 comp rx=0 tx=0).stop 2024-09-16T09:16:14.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 shutdown_connections 2024-09-16T09:16:14.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7fae5807ae10 0x7fae5807d2d0 unknown :-1 s=CLOSED pgs=3938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6c107f70 0x7fae6c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae6c1075f0 0x7fae6c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 --2- 172.21.15.111:0/2219648310 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae6c101bf0 0x7fae6c197620 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 >> 172.21.15.111:0/2219648310 conn(0x7fae6c0fd630 msgr2=0x7fae6c0feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:14.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 shutdown_connections 2024-09-16T09:16:14.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.521+0000 7fae74589640 1 -- 172.21.15.111:0/2219648310 wait complete. 2024-09-16T09:16:14.531 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool application enable 4b166c41-77f3-458e-9e25-c5fe356364a4 rados 2024-09-16T09:16:14.615 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.612+0000 7f682a99d640 1 -- 172.21.15.111:0/570524234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6824103a40 msgr2=0x7f6824103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:14.615 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.612+0000 7f682a99d640 1 --2- 172.21.15.111:0/570524234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6824103a40 0x7f6824103e20 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f6820009900 tx=0x7f682002a3f0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.615 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.612+0000 7f682a99d640 1 -- 172.21.15.111:0/570524234 shutdown_connections 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.612+0000 7f682a99d640 1 --2- 172.21.15.111:0/570524234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f682410ce90 0x7f682410f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.612+0000 7f682a99d640 1 --2- 172.21.15.111:0/570524234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824104360 0x7f682410c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.612+0000 7f682a99d640 1 --2- 172.21.15.111:0/570524234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6824103a40 0x7f6824103e20 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.612+0000 7f682a99d640 1 -- 172.21.15.111:0/570524234 >> 172.21.15.111:0/570524234 conn(0x7f68240fd470 msgr2=0x7f68240ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.612+0000 7f682a99d640 1 -- 172.21.15.111:0/570524234 shutdown_connections 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.612+0000 7f682a99d640 1 -- 172.21.15.111:0/570524234 wait complete. 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a99d640 1 Processor -- start 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a99d640 1 -- start start 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a99d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824103a40 0x7f68241016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a99d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6824104360 0x7f6824101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a99d640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f682410ce90 0x7f6824102160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a99d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6824111ff0 con 0x7f6824104360 2024-09-16T09:16:14.616 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a99d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6824111e70 con 0x7f6824103a40 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a99d640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f6824112170 con 0x7f682410ce90 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682919a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6824104360 0x7f6824101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a19c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f682410ce90 0x7f6824102160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682919a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6824104360 0x7f6824101c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50118/0 (socket says 172.21.15.111:50118) 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a19c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f682410ce90 0x7f6824102160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:55194/0 (socket says 172.21.15.111:55194) 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682919a640 1 -- 172.21.15.111:0/4071713201 learned_addr learned my addr 172.21.15.111:0/4071713201 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682999b640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824103a40 0x7f68241016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682919a640 1 -- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f682410ce90 msgr2=0x7f6824102160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682919a640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f682410ce90 0x7f6824102160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682919a640 1 -- 172.21.15.111:0/4071713201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824103a40 msgr2=0x7f68241016e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682919a640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824103a40 0x7f68241016e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:14.617 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682919a640 1 -- 172.21.15.111:0/4071713201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68241a69a0 con 0x7f6824104360 2024-09-16T09:16:14.618 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682a19c640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f682410ce90 0x7f6824102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:16:14.618 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682999b640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824103a40 0x7f68241016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T09:16:14.618 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.613+0000 7f682919a640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6824104360 0x7f6824101c20 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f681400c970 tx=0x7f681400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:14.618 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.614+0000 7f681affd640 1 -- 172.21.15.111:0/4071713201 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6814007bb0 con 0x7f6824104360 2024-09-16T09:16:14.618 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.614+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68241a6c90 con 0x7f6824104360 2024-09-16T09:16:14.618 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.614+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68241a71d0 con 0x7f6824104360 2024-09-16T09:16:14.618 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.614+0000 7f681affd640 1 -- 172.21.15.111:0/4071713201 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f6814007d50 con 0x7f6824104360 2024-09-16T09:16:14.618 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.614+0000 7f681affd640 1 -- 172.21.15.111:0/4071713201 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681400f450 con 0x7f6824104360 2024-09-16T09:16:14.619 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.615+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6824103e20 con 0x7f6824104360 2024-09-16T09:16:14.619 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.615+0000 7f681affd640 1 -- 172.21.15.111:0/4071713201 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f6814018480 con 0x7f6824104360 2024-09-16T09:16:14.619 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.616+0000 7f681affd640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f681007ae10 0x7f681007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:14.619 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.616+0000 7f681affd640 1 -- 172.21.15.111:0/4071713201 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(754..754 src has 1..754) ==== 9438+0+0 (secure 0 0 0) 0x7f681409ef50 con 0x7f6824104360 2024-09-16T09:16:14.619 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.616+0000 7f682999b640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f681007ae10 0x7f681007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:14.619 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.616+0000 7f682999b640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f681007ae10 0x7f681007d2d0 secure :-1 s=READY pgs=3939 cs=0 l=1 rev1=1 crypto rx=0x7f68200057c0 tx=0x7f6820000bc0 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:14.622 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.621+0000 7f681affd640 1 -- 172.21.15.111:0/4071713201 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f68140699d0 con 0x7f6824104360 2024-09-16T09:16:14.799 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:14.798+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"} v 0) -- 0x7f6824107930 con 0x7f6824104360 2024-09-16T09:16:15.410 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:15.409+0000 7f681affd640 1 -- 172.21.15.111:0/4071713201 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"}]=0 enabled application 'rados' on pool '4b166c41-77f3-458e-9e25-c5fe356364a4' v755) ==== 213+0+0 (secure 0 0 0) 0x7f681406e880 con 0x7f6824104360 2024-09-16T09:16:15.527 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:15.526+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"} v 0) -- 0x7f6824107b80 con 0x7f6824104360 2024-09-16T09:16:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12}]': finished 2024-09-16T09:16:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[40056]: osdmap e754: 8 total, 8 up, 8 in 2024-09-16T09:16:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2219648310' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:15.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"} : dispatch 2024-09-16T09:16:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12}]': finished 2024-09-16T09:16:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[29803]: osdmap e754: 8 total, 8 up, 8 in 2024-09-16T09:16:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2219648310' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:15.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:15 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"} : dispatch 2024-09-16T09:16:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12}]': finished 2024-09-16T09:16:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:15 smithi114 ceph-mon[33315]: osdmap e754: 8 total, 8 up, 8 in 2024-09-16T09:16:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2219648310' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:15 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pg_num": 12} : dispatch 2024-09-16T09:16:15.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:15 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"} : dispatch 2024-09-16T09:16:16.416 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.415+0000 7f681affd640 1 -- 172.21.15.111:0/4071713201 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"}]=0 enabled application 'rados' on pool '4b166c41-77f3-458e-9e25-c5fe356364a4' v756) ==== 213+0+0 (secure 0 0 0) 0x7f681409c680 con 0x7f6824104360 2024-09-16T09:16:16.417 INFO:tasks.workunit.client.0.smithi111.stderr:enabled application 'rados' on pool '4b166c41-77f3-458e-9e25-c5fe356364a4' 2024-09-16T09:16:16.421 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f681007ae10 msgr2=0x7f681007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:16.421 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f681007ae10 0x7f681007d2d0 secure :-1 s=READY pgs=3939 cs=0 l=1 rev1=1 crypto rx=0x7f68200057c0 tx=0x7f6820000bc0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.421 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6824104360 msgr2=0x7f6824101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:16.421 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6824104360 0x7f6824101c20 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f681400c970 tx=0x7f681400cd70 comp rx=0 tx=0).stop 2024-09-16T09:16:16.422 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 shutdown_connections 2024-09-16T09:16:16.422 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f681007ae10 0x7f681007d2d0 unknown :-1 s=CLOSED pgs=3939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.422 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f682410ce90 0x7f6824102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.422 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6824104360 0x7f6824101c20 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.422 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 --2- 172.21.15.111:0/4071713201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824103a40 0x7f68241016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.422 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.420+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 >> 172.21.15.111:0/4071713201 conn(0x7f68240fd470 msgr2=0x7f68240fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:16.422 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.421+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 shutdown_connections 2024-09-16T09:16:16.422 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.421+0000 7f682a99d640 1 -- 172.21.15.111:0/4071713201 wait complete. 2024-09-16T09:16:16.433 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool set-quota 4b166c41-77f3-458e-9e25-c5fe356364a4 max_objects 10 2024-09-16T09:16:16.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.518+0000 7ff9bc02a640 1 -- 172.21.15.111:0/1024284884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b4102d00 msgr2=0x7ff9b41030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:16.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.518+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/1024284884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b4102d00 0x7ff9b41030e0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009900 tx=0x7ff9b002a410 comp rx=0 tx=0).stop 2024-09-16T09:16:16.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.518+0000 7ff9bc02a640 1 -- 172.21.15.111:0/1024284884 shutdown_connections 2024-09-16T09:16:16.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.518+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/1024284884 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9b4109080 0x7ff9b410f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.518+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/1024284884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b4108700 0x7ff9b4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.518+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/1024284884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b4102d00 0x7ff9b41030e0 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.518+0000 7ff9bc02a640 1 -- 172.21.15.111:0/1024284884 >> 172.21.15.111:0/1024284884 conn(0x7ff9b40fe740 msgr2=0x7ff9b4100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:16.522 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.518+0000 7ff9bc02a640 1 -- 172.21.15.111:0/1024284884 shutdown_connections 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.518+0000 7ff9bc02a640 1 -- 172.21.15.111:0/1024284884 wait complete. 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9bc02a640 1 Processor -- start 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9bc02a640 1 -- start start 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9bc02a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b4102d00 0x7ff9b4198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9bc02a640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9b4108700 0x7ff9b4198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9bc02a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b4109080 0x7ff9b419e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9bc02a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b4112210 con 0x7ff9b4102d00 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9bc02a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b4112090 con 0x7ff9b4109080 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9bc02a640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b4112390 con 0x7ff9b4108700 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9b95c5640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9b4108700 0x7ff9b4198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9b9dc6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b4102d00 0x7ff9b4198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9b95c5640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9b4108700 0x7ff9b4198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:55214/0 (socket says 172.21.15.111:55214) 2024-09-16T09:16:16.523 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.519+0000 7ff9b95c5640 1 -- 172.21.15.111:0/2611013119 learned_addr learned my addr 172.21.15.111:0/2611013119 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9ba5c7640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b4109080 0x7ff9b419e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9b9dc6640 1 -- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9b4108700 msgr2=0x7ff9b4198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9b9dc6640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9b4108700 0x7ff9b4198da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9b9dc6640 1 -- 172.21.15.111:0/2611013119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b4109080 msgr2=0x7ff9b419e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9b9dc6640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b4109080 0x7ff9b419e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9b9dc6640 1 -- 172.21.15.111:0/2611013119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b419e900 con 0x7ff9b4102d00 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9b95c5640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9b4108700 0x7ff9b4198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9b9dc6640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b4102d00 0x7ff9b4198860 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7ff9b002a930 tx=0x7ff9b0032af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9aaffd640 1 -- 172.21.15.111:0/2611013119 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b003f030 con 0x7ff9b4102d00 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b419eb90 con 0x7ff9b4102d00 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9aaffd640 1 -- 172.21.15.111:0/2611013119 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7ff9b0003a40 con 0x7ff9b4102d00 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9aaffd640 1 -- 172.21.15.111:0/2611013119 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0035070 con 0x7ff9b4102d00 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.520+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b41a52b0 con 0x7ff9b4102d00 2024-09-16T09:16:16.524 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.521+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b41030e0 con 0x7ff9b4102d00 2024-09-16T09:16:16.525 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.522+0000 7ff9aaffd640 1 -- 172.21.15.111:0/2611013119 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7ff9b0042030 con 0x7ff9b4102d00 2024-09-16T09:16:16.525 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.522+0000 7ff9aaffd640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7ff9a007ae10 0x7ff9a007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:16:16.525 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.522+0000 7ff9b95c5640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7ff9a007ae10 0x7ff9a007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:16:16.525 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.522+0000 7ff9aaffd640 1 -- 172.21.15.111:0/2611013119 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(756..756 src has 1..756) ==== 9451+0+0 (secure 0 0 0) 0x7ff9b00beff0 con 0x7ff9b4102d00 2024-09-16T09:16:16.525 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.523+0000 7ff9b95c5640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7ff9a007ae10 0x7ff9a007d2d0 secure :-1 s=READY pgs=3940 cs=0 l=1 rev1=1 crypto rx=0x7ff9b41021b0 tx=0x7ff9a400b4a0 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:16:16.527 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.527+0000 7ff9aaffd640 1 -- 172.21.15.111:0/2611013119 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7ff9b0089a60 con 0x7ff9b4102d00 2024-09-16T09:16:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:16 smithi114 ceph-mon[33315]: pgmap v1196: 188 pgs: 12 unknown, 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-16T09:16:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:16 smithi114 ceph-mon[33315]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:16:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"}]': finished 2024-09-16T09:16:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:16 smithi114 ceph-mon[33315]: osdmap e755: 8 total, 8 up, 8 in 2024-09-16T09:16:16.664 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:16 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"} : dispatch 2024-09-16T09:16:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[29803]: pgmap v1196: 188 pgs: 12 unknown, 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-16T09:16:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[29803]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:16:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"}]': finished 2024-09-16T09:16:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[29803]: osdmap e755: 8 total, 8 up, 8 in 2024-09-16T09:16:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"} : dispatch 2024-09-16T09:16:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[40056]: pgmap v1196: 188 pgs: 12 unknown, 176 active+clean; 733 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-16T09:16:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[40056]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:16:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"}]': finished 2024-09-16T09:16:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[40056]: osdmap e755: 8 total, 8 up, 8 in 2024-09-16T09:16:16.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:16 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"} : dispatch 2024-09-16T09:16:16.709 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:16.708+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"} v 0) -- 0x7ff9b410ae60 con 0x7ff9b4102d00 2024-09-16T09:16:17.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"}]': finished 2024-09-16T09:16:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[29803]: osdmap e756: 8 total, 8 up, 8 in 2024-09-16T09:16:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:16:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:17.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:16:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"}]': finished 2024-09-16T09:16:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[40056]: osdmap e756: 8 total, 8 up, 8 in 2024-09-16T09:16:17.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:16:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:17.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:17 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:16:17.707 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:17.706+0000 7ff9aaffd640 1 -- 172.21.15.111:0/2611013119 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 4b166c41-77f3-458e-9e25-c5fe356364a4 v757) ==== 223+0+0 (secure 0 0 0) 0x7ff9b008e910 con 0x7ff9b4102d00 2024-09-16T09:16:17.819 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:17.818+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"} v 0) -- 0x7ff9b410aa70 con 0x7ff9b4102d00 2024-09-16T09:16:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4071713201' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "app": "rados"}]': finished 2024-09-16T09:16:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:17 smithi114 ceph-mon[33315]: osdmap e756: 8 total, 8 up, 8 in 2024-09-16T09:16:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:16:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:17 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:16:18.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:18 smithi111 ceph-mon[29803]: pgmap v1199: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.7 KiB/s wr, 1 op/s 2024-09-16T09:16:18.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:16:18.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:18 smithi111 ceph-mon[29803]: osdmap e757: 8 total, 8 up, 8 in 2024-09-16T09:16:18.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:18 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:16:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:18 smithi111 ceph-mon[40056]: pgmap v1199: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.7 KiB/s wr, 1 op/s 2024-09-16T09:16:18.677 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:16:18.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:18 smithi111 ceph-mon[40056]: osdmap e757: 8 total, 8 up, 8 in 2024-09-16T09:16:18.678 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:18 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:16:18.709 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.708+0000 7ff9aaffd640 1 -- 172.21.15.111:0/2611013119 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 4b166c41-77f3-458e-9e25-c5fe356364a4 v758) ==== 223+0+0 (secure 0 0 0) 0x7ff9b00bc710 con 0x7ff9b4102d00 2024-09-16T09:16:18.709 INFO:tasks.workunit.client.0.smithi111.stderr:set-quota max_objects = 10 for pool 4b166c41-77f3-458e-9e25-c5fe356364a4 2024-09-16T09:16:18.714 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.712+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7ff9a007ae10 msgr2=0x7ff9a007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:18.714 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.712+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7ff9a007ae10 0x7ff9a007d2d0 secure :-1 s=READY pgs=3940 cs=0 l=1 rev1=1 crypto rx=0x7ff9b41021b0 tx=0x7ff9a400b4a0 comp rx=0 tx=0).stop 2024-09-16T09:16:18.714 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.712+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b4102d00 msgr2=0x7ff9b4198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:16:18.714 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.712+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b4102d00 0x7ff9b4198860 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7ff9b002a930 tx=0x7ff9b0032af0 comp rx=0 tx=0).stop 2024-09-16T09:16:18.714 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.712+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 shutdown_connections 2024-09-16T09:16:18.714 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.712+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7ff9a007ae10 0x7ff9a007d2d0 unknown :-1 s=CLOSED pgs=3940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:18.714 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.712+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9b4109080 0x7ff9b419e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:18.715 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.712+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9b4108700 0x7ff9b4198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:18.715 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.712+0000 7ff9bc02a640 1 --2- 172.21.15.111:0/2611013119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9b4102d00 0x7ff9b4198860 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:16:18.715 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.713+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 >> 172.21.15.111:0/2611013119 conn(0x7ff9b40fe740 msgr2=0x7ff9b40ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:16:18.715 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.713+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 shutdown_connections 2024-09-16T09:16:18.715 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:16:18.713+0000 7ff9bc02a640 1 -- 172.21.15.111:0/2611013119 wait complete. 2024-09-16T09:16:18.725 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 30 2024-09-16T09:16:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:18 smithi114 ceph-mon[33315]: pgmap v1199: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.7 KiB/s wr, 1 op/s 2024-09-16T09:16:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:16:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:18 smithi114 ceph-mon[33315]: osdmap e757: 8 total, 8 up, 8 in 2024-09-16T09:16:18.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:18 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"} : dispatch 2024-09-16T09:16:20.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:16:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:19 smithi114 ceph-mon[33315]: osdmap e758: 8 total, 8 up, 8 in 2024-09-16T09:16:20.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:19 smithi114 ceph-mon[33315]: pgmap v1202: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 1 op/s 2024-09-16T09:16:20.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:16:20.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:19 smithi111 ceph-mon[40056]: osdmap e758: 8 total, 8 up, 8 in 2024-09-16T09:16:20.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:19 smithi111 ceph-mon[40056]: pgmap v1202: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 1 op/s 2024-09-16T09:16:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2611013119' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "field": "max_objects", "val": "10"}]': finished 2024-09-16T09:16:20.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:19 smithi111 ceph-mon[29803]: osdmap e758: 8 total, 8 up, 8 in 2024-09-16T09:16:20.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:19 smithi111 ceph-mon[29803]: pgmap v1202: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 1 op/s 2024-09-16T09:16:22.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:21 smithi114 ceph-mon[33315]: pgmap v1203: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 4.9 KiB/s wr, 1 op/s 2024-09-16T09:16:22.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:21 smithi111 ceph-mon[29803]: pgmap v1203: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 4.9 KiB/s wr, 1 op/s 2024-09-16T09:16:22.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:21 smithi111 ceph-mon[40056]: pgmap v1203: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 4.9 KiB/s wr, 1 op/s 2024-09-16T09:16:22.664 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:16:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:16:23.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:22 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:23.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:22 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:23.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:22 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:23.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:16:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:16:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:16:24.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:23 smithi114 ceph-mon[33315]: pgmap v1204: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 4.1 KiB/s wr, 1 op/s 2024-09-16T09:16:24.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:23 smithi114 ceph-mon[33315]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:16:24.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:23 smithi111 ceph-mon[29803]: pgmap v1204: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 4.1 KiB/s wr, 1 op/s 2024-09-16T09:16:24.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:23 smithi111 ceph-mon[29803]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:16:24.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:23 smithi111 ceph-mon[40056]: pgmap v1204: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 4.1 KiB/s wr, 1 op/s 2024-09-16T09:16:24.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:23 smithi111 ceph-mon[40056]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T09:16:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:25 smithi114 ceph-mon[33315]: pgmap v1205: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:16:26.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:25 smithi111 ceph-mon[29803]: pgmap v1205: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:16:26.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:25 smithi111 ceph-mon[40056]: pgmap v1205: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:16:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:27 smithi114 ceph-mon[33315]: pgmap v1206: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T09:16:28.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:27 smithi111 ceph-mon[29803]: pgmap v1206: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T09:16:28.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:27 smithi111 ceph-mon[40056]: pgmap v1206: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-16T09:16:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:29 smithi114 ceph-mon[33315]: pgmap v1207: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-09-16T09:16:30.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:29 smithi111 ceph-mon[29803]: pgmap v1207: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-09-16T09:16:30.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:29 smithi111 ceph-mon[40056]: pgmap v1207: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-09-16T09:16:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:31 smithi114 ceph-mon[33315]: pgmap v1208: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:31 smithi111 ceph-mon[29803]: pgmap v1208: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:32.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:31 smithi111 ceph-mon[40056]: pgmap v1208: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:32.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:32.853 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:16:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:16:33.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:32 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:33.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:32 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:33.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:32 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:16:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:16:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:16:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:33 smithi114 ceph-mon[33315]: pgmap v1209: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:33 smithi111 ceph-mon[29803]: pgmap v1209: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:33 smithi111 ceph-mon[40056]: pgmap v1209: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:35 smithi114 ceph-mon[33315]: pgmap v1210: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:35 smithi111 ceph-mon[29803]: pgmap v1210: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:36.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:35 smithi111 ceph-mon[40056]: pgmap v1210: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:37 smithi114 ceph-mon[33315]: pgmap v1211: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:38.180 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:37 smithi111 ceph-mon[29803]: pgmap v1211: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:38.180 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:37 smithi111 ceph-mon[40056]: pgmap v1211: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:39 smithi114 ceph-mon[33315]: pgmap v1212: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:39 smithi111 ceph-mon[29803]: pgmap v1212: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:40.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:39 smithi111 ceph-mon[40056]: pgmap v1212: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:42.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:41 smithi114 ceph-mon[33315]: pgmap v1213: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:41 smithi111 ceph-mon[29803]: pgmap v1213: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:41 smithi111 ceph-mon[40056]: pgmap v1213: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:42.868 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:16:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:16:43.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:42 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:43.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:42 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:43.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:42 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:16:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:16:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:16:44.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:43 smithi114 ceph-mon[33315]: pgmap v1214: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:44.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:43 smithi111 ceph-mon[29803]: pgmap v1214: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:43 smithi111 ceph-mon[40056]: pgmap v1214: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:45 smithi114 ceph-mon[33315]: pgmap v1215: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:45 smithi111 ceph-mon[29803]: pgmap v1215: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:46.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:45 smithi111 ceph-mon[40056]: pgmap v1215: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:47.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:47.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:16:48.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:47 smithi114 ceph-mon[33315]: pgmap v1216: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:47 smithi111 ceph-mon[29803]: pgmap v1216: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:48.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:47 smithi111 ceph-mon[40056]: pgmap v1216: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:16:48.727 INFO:tasks.workunit.client.0.smithi111.stderr:++ seq 1 10 2024-09-16T09:16:48.728 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:48.728 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj1 /etc/passwd 2024-09-16T09:16:48.770 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:48.770 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj2 /etc/passwd 2024-09-16T09:16:48.809 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:48.809 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj3 /etc/passwd 2024-09-16T09:16:48.846 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:48.846 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj4 /etc/passwd 2024-09-16T09:16:48.883 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:48.883 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj5 /etc/passwd 2024-09-16T09:16:48.919 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:48.919 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj6 /etc/passwd 2024-09-16T09:16:48.955 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:48.955 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj7 /etc/passwd 2024-09-16T09:16:48.990 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:48.991 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj8 /etc/passwd 2024-09-16T09:16:49.026 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:49.026 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj9 /etc/passwd 2024-09-16T09:16:49.061 INFO:tasks.workunit.client.0.smithi111.stderr:+ for f in `seq 1 10` 2024-09-16T09:16:49.061 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p 4b166c41-77f3-458e-9e25-c5fe356364a4 put obj10 /etc/passwd 2024-09-16T09:16:49.097 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 30 2024-09-16T09:16:50.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:49 smithi114 ceph-mon[33315]: pgmap v1217: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:49 smithi111 ceph-mon[29803]: pgmap v1217: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:50.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:49 smithi111 ceph-mon[40056]: pgmap v1217: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:51 smithi114 ceph-mon[33315]: pgmap v1218: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:51 smithi111 ceph-mon[29803]: pgmap v1218: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:52.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:51 smithi111 ceph-mon[40056]: pgmap v1218: 188 pgs: 188 active+clean; 759 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:16:52.899 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:16:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:16:53.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:52 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:53.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:52 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:52 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:16:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:16:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:16:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:16:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:53 smithi114 ceph-mon[33315]: pgmap v1219: 188 pgs: 188 active+clean; 884 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 2 op/s 2024-09-16T09:16:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:53 smithi114 ceph-mon[33315]: pool '4b166c41-77f3-458e-9e25-c5fe356364a4' is full (reached quota's max_objects: 10) 2024-09-16T09:16:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:53 smithi114 ceph-mon[33315]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:16:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:53 smithi114 ceph-mon[33315]: osdmap e759: 8 total, 8 up, 8 in 2024-09-16T09:16:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:53 smithi111 ceph-mon[29803]: pgmap v1219: 188 pgs: 188 active+clean; 884 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 2 op/s 2024-09-16T09:16:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:53 smithi111 ceph-mon[29803]: pool '4b166c41-77f3-458e-9e25-c5fe356364a4' is full (reached quota's max_objects: 10) 2024-09-16T09:16:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:53 smithi111 ceph-mon[29803]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:16:54.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:53 smithi111 ceph-mon[29803]: osdmap e759: 8 total, 8 up, 8 in 2024-09-16T09:16:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:53 smithi111 ceph-mon[40056]: pgmap v1219: 188 pgs: 188 active+clean; 884 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 2 op/s 2024-09-16T09:16:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:53 smithi111 ceph-mon[40056]: pool '4b166c41-77f3-458e-9e25-c5fe356364a4' is full (reached quota's max_objects: 10) 2024-09-16T09:16:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:53 smithi111 ceph-mon[40056]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-16T09:16:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:53 smithi111 ceph-mon[40056]: osdmap e759: 8 total, 8 up, 8 in 2024-09-16T09:16:56.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:55 smithi114 ceph-mon[33315]: pgmap v1221: 188 pgs: 188 active+clean; 884 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:16:56.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:55 smithi111 ceph-mon[29803]: pgmap v1221: 188 pgs: 188 active+clean; 884 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:16:56.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:55 smithi111 ceph-mon[40056]: pgmap v1221: 188 pgs: 188 active+clean; 884 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:16:58.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:57 smithi114 ceph-mon[33315]: pgmap v1222: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:16:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:57 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:16:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:57 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:16:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:57 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:16:58.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:57 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:16:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[29803]: pgmap v1222: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:16:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:16:58.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:16:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:16:58.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:16:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[40056]: pgmap v1222: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:16:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-16T09:16:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-16T09:16:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-16T09:16:58.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:57 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:17:00.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:16:59 smithi114 ceph-mon[33315]: pgmap v1223: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:17:00.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:16:59 smithi111 ceph-mon[40056]: pgmap v1223: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:17:00.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:16:59 smithi111 ceph-mon[29803]: pgmap v1223: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:17:02.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:01 smithi114 ceph-mon[33315]: pgmap v1224: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:17:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:17:02.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:01 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:01 smithi111 ceph-mon[29803]: pgmap v1224: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:17:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:17:02.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:01 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:02.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:01 smithi111 ceph-mon[40056]: pgmap v1224: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-16T09:17:02.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' 2024-09-16T09:17:02.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:01 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:02.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:17:02 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:17:03.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:02 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:03.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:02 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:03.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:02 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:03.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:17:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:17:04.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:03 smithi111 ceph-mon[29803]: pgmap v1225: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:17:04.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:03 smithi111 ceph-mon[40056]: pgmap v1225: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:17:04.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:03 smithi114 ceph-mon[33315]: pgmap v1225: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-16T09:17:06.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:05 smithi111 ceph-mon[29803]: pgmap v1226: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-16T09:17:06.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:05 smithi111 ceph-mon[40056]: pgmap v1226: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-16T09:17:06.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:05 smithi114 ceph-mon[33315]: pgmap v1226: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-16T09:17:08.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:07 smithi111 ceph-mon[29803]: pgmap v1227: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:08.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:07 smithi111 ceph-mon[40056]: pgmap v1227: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:08.413 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:07 smithi114 ceph-mon[33315]: pgmap v1227: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:10.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:09 smithi114 ceph-mon[33315]: pgmap v1228: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:10.426 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:09 smithi111 ceph-mon[29803]: pgmap v1228: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:10.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:09 smithi111 ceph-mon[40056]: pgmap v1228: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:12.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:11 smithi114 ceph-mon[33315]: pgmap v1229: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:12.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:11 smithi111 ceph-mon[40056]: pgmap v1229: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:12.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:11 smithi111 ceph-mon[29803]: pgmap v1229: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:12.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:17:12 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:17:13.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:12 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:13.426 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:12 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:13.427 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:12 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:13.427 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:13 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:17:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:17:14.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:14 smithi111 ceph-mon[29803]: pgmap v1230: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:14.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:14 smithi111 ceph-mon[40056]: pgmap v1230: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:14.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:14 smithi114 ceph-mon[33315]: pgmap v1230: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:16.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:16 smithi111 ceph-mon[29803]: pgmap v1231: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:16.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:16 smithi111 ceph-mon[40056]: pgmap v1231: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:16.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:16 smithi114 ceph-mon[33315]: pgmap v1231: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:17.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:17 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:17.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:17 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:17.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:17 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:18.913 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:18 smithi114 ceph-mon[33315]: pgmap v1232: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:18.929 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:18 smithi111 ceph-mon[40056]: pgmap v1232: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:18.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:18 smithi111 ceph-mon[29803]: pgmap v1232: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:19.098 INFO:tasks.workunit.client.0.smithi111.stderr:+ rados -p aa242ffd-5c69-42f2-92b2-30d0602f301c put threemore /etc/passwd 2024-09-16T09:17:19.140 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool set-quota aa242ffd-5c69-42f2-92b2-30d0602f301c max_bytes 0 2024-09-16T09:17:19.229 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.225+0000 7f88630bd640 1 -- 172.21.15.111:0/560122006 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c109080 msgr2=0x7f885c10f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:19.229 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.225+0000 7f88630bd640 1 --2- 172.21.15.111:0/560122006 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c109080 0x7f885c10f880 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f8854009900 tx=0x7f885402a410 comp rx=0 tx=0).stop 2024-09-16T09:17:19.229 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.225+0000 7f88630bd640 1 -- 172.21.15.111:0/560122006 shutdown_connections 2024-09-16T09:17:19.229 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.225+0000 7f88630bd640 1 --2- 172.21.15.111:0/560122006 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c109080 0x7f885c10f880 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:19.229 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.225+0000 7f88630bd640 1 --2- 172.21.15.111:0/560122006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c108700 0x7f885c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:19.229 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.225+0000 7f88630bd640 1 --2- 172.21.15.111:0/560122006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c102d00 0x7f885c1030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.225+0000 7f88630bd640 1 -- 172.21.15.111:0/560122006 >> 172.21.15.111:0/560122006 conn(0x7f885c0fe740 msgr2=0x7f885c100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.225+0000 7f88630bd640 1 -- 172.21.15.111:0/560122006 shutdown_connections 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.225+0000 7f88630bd640 1 -- 172.21.15.111:0/560122006 wait complete. 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f88630bd640 1 Processor -- start 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f88630bd640 1 -- start start 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f88630bd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c102d00 0x7f885c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f88630bd640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c108700 0x7f885c198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f88630bd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c109080 0x7f885c19e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f88630bd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f885c112220 con 0x7f885c109080 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f88630bd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1120a0 con 0x7f885c102d00 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f88630bd640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f885c1123a0 con 0x7f885c108700 2024-09-16T09:17:19.230 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f885bfff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c108700 0x7f885c198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f885bfff640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c108700 0x7f885c198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:40028/0 (socket says 172.21.15.111:40028) 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f885bfff640 1 -- 172.21.15.111:0/2124482270 learned_addr learned my addr 172.21.15.111:0/2124482270 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.226+0000 7f886165a640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c109080 0x7f885c19e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f8860e59640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c102d00 0x7f885c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f885bfff640 1 -- 172.21.15.111:0/2124482270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c102d00 msgr2=0x7f885c1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f885bfff640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c102d00 0x7f885c1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f885bfff640 1 -- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c109080 msgr2=0x7f885c19e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f885bfff640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c109080 0x7f885c19e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f885bfff640 1 -- 172.21.15.111:0/2124482270 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885c19e880 con 0x7f885c108700 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f885bfff640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c108700 0x7f885c198d20 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f884400d950 tx=0x7f884400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8844017030 con 0x7f885c108700 2024-09-16T09:17:19.231 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f8844004020 con 0x7f885c108700 2024-09-16T09:17:19.232 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885c19eb70 con 0x7f885c108700 2024-09-16T09:17:19.232 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8844005430 con 0x7f885c108700 2024-09-16T09:17:19.232 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.227+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f885c1a5230 con 0x7f885c108700 2024-09-16T09:17:19.232 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.228+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f885c1030e0 con 0x7f885c108700 2024-09-16T09:17:19.233 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.229+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f8844009a30 con 0x7f885c108700 2024-09-16T09:17:19.233 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.229+0000 7f8859ffb640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f884807ae10 0x7f884807d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:19.233 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.229+0000 7f8860e59640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f884807ae10 0x7f884807d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:19.233 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.229+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(759..759 src has 1..759) ==== 9451+0+0 (secure 0 0 0) 0x7f884409e5d0 con 0x7f885c108700 2024-09-16T09:17:19.233 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.230+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=760}) -- 0x7f88480870b0 con 0x7f885c108700 2024-09-16T09:17:19.233 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.230+0000 7f8860e59640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f884807ae10 0x7f884807d2d0 secure :-1 s=READY pgs=3952 cs=0 l=1 rev1=1 crypto rx=0x7f884c005fd0 tx=0x7f884c009500 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:17:19.235 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.234+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f8844069040 con 0x7f885c108700 2024-09-16T09:17:19.414 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.413+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} v 0) -- 0x7f885c10ae60 con 0x7f885c108700 2024-09-16T09:17:19.438 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.437+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 7 ==== osd_map(760..760 src has 1..760) ==== 628+0+0 (secure 0 0 0) 0x7f88440d5190 con 0x7f885c108700 2024-09-16T09:17:19.438 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.437+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=761}) -- 0x7f8848087c60 con 0x7f885c108700 2024-09-16T09:17:19.442 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.441+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v760) ==== 217+0+0 (secure 0 0 0) 0x7f884406def0 con 0x7f885c108700 2024-09-16T09:17:19.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:19.547+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} v 0) -- 0x7f885c10a8d0 con 0x7f885c108700 2024-09-16T09:17:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:19 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2124482270' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:19.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:19 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:19 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2124482270' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:19.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:19 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:19.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:19 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2124482270' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:19.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:19 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:20.440 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.438+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 9 ==== osd_map(761..761 src has 1..761) ==== 628+0+0 (secure 0 0 0) 0x7f884406b070 con 0x7f885c108700 2024-09-16T09:17:20.440 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.438+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=762}) -- 0x7f8848088260 con 0x7f885c108700 2024-09-16T09:17:20.443 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.442+0000 7f8859ffb640 1 -- 172.21.15.111:0/2124482270 <== mon.2 v2:172.21.15.111:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v761) ==== 217+0+0 (secure 0 0 0) 0x7f884409bcf0 con 0x7f885c108700 2024-09-16T09:17:20.443 INFO:tasks.workunit.client.0.smithi111.stderr:set-quota max_bytes = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f884807ae10 msgr2=0x7f884807d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f884807ae10 0x7f884807d2d0 secure :-1 s=READY pgs=3952 cs=0 l=1 rev1=1 crypto rx=0x7f884c005fd0 tx=0x7f884c009500 comp rx=0 tx=0).stop 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c108700 msgr2=0x7f885c198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c108700 0x7f885c198d20 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f884400d950 tx=0x7f884400de10 comp rx=0 tx=0).stop 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 shutdown_connections 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f884807ae10 0x7f884807d2d0 unknown :-1 s=CLOSED pgs=3952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f885c109080 0x7f885c19e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f885c108700 0x7f885c198d20 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 --2- 172.21.15.111:0/2124482270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c102d00 0x7f885c1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.446+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 >> 172.21.15.111:0/2124482270 conn(0x7f885c0fe740 msgr2=0x7f885c106b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:17:20.448 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.447+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 shutdown_connections 2024-09-16T09:17:20.449 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.447+0000 7f88630bd640 1 -- 172.21.15.111:0/2124482270 wait complete. 2024-09-16T09:17:20.459 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool set-quota aa242ffd-5c69-42f2-92b2-30d0602f301c max_objects 0 2024-09-16T09:17:20.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 -- 172.21.15.111:0/1458079085 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800107f60 msgr2=0x7f880010e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:20.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 --2- 172.21.15.111:0/1458079085 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800107f60 0x7f880010e760 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f87f8009ca0 tx=0x7f87f802a8c0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 -- 172.21.15.111:0/1458079085 shutdown_connections 2024-09-16T09:17:20.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 --2- 172.21.15.111:0/1458079085 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800107f60 0x7f880010e760 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 --2- 172.21.15.111:0/1458079085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88001075e0 0x7f8800107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.547 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 --2- 172.21.15.111:0/1458079085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8800101be0 0x7f8800101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 -- 172.21.15.111:0/1458079085 >> 172.21.15.111:0/1458079085 conn(0x7f88000fd640 msgr2=0x7f88000ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 -- 172.21.15.111:0/1458079085 shutdown_connections 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 -- 172.21.15.111:0/1458079085 wait complete. 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.544+0000 7f8804a9b640 1 Processor -- start 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f8804a9b640 1 -- start start 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f8804a9b640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800101be0 0x7f880010f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f8804a9b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88001075e0 0x7f880010f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f8804a9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8800107f60 0x7f880010fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f8804a9b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f88001121f0 con 0x7f88001075e0 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f8804a9b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8800112070 con 0x7f8800107f60 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f8804a9b640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f8800112370 con 0x7f8800101be0 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fe59c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800101be0 0x7f880010f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:20.548 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87f7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88001075e0 0x7f880010f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fe59c640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800101be0 0x7f880010f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:37238/0 (socket says 172.21.15.111:37238) 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87f7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88001075e0 0x7f880010f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45674/0 (socket says 172.21.15.111:45674) 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fe59c640 1 -- 172.21.15.111:0/539471565 learned_addr learned my addr 172.21.15.111:0/539471565 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fed9d640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8800107f60 0x7f880010fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fe59c640 1 -- 172.21.15.111:0/539471565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8800107f60 msgr2=0x7f880010fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fe59c640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8800107f60 0x7f880010fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fe59c640 1 -- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88001075e0 msgr2=0x7f880010f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fe59c640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88001075e0 0x7f880010f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fe59c640 1 -- 172.21.15.111:0/539471565 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88001a6b00 con 0x7f8800101be0 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87f7fff640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88001075e0 0x7f880010f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fed9d640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8800107f60 0x7f880010fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87fe59c640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800101be0 0x7f880010f270 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f87f000ea60 tx=0x7f87f000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:17:20.549 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f000cd20 con 0x7f8800101be0 2024-09-16T09:17:20.550 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f87f0004470 con 0x7f8800101be0 2024-09-16T09:17:20.550 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88001a6df0 con 0x7f8800101be0 2024-09-16T09:17:20.550 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.545+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f0003e60 con 0x7f8800101be0 2024-09-16T09:17:20.550 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.546+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88001a7300 con 0x7f8800101be0 2024-09-16T09:17:20.551 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.546+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8800101fc0 con 0x7f8800101be0 2024-09-16T09:17:20.551 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.547+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f87f0004000 con 0x7f8800101be0 2024-09-16T09:17:20.551 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.547+0000 7f87f77fe640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f87ec07ae10 0x7f87ec07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:20.551 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.548+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(761..761 src has 1..761) ==== 9451+0+0 (secure 0 0 0) 0x7f87f009efa0 con 0x7f8800101be0 2024-09-16T09:17:20.551 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.548+0000 7f87f7fff640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f87ec07ae10 0x7f87ec07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:20.551 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.548+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=762}) -- 0x7f87ec087130 con 0x7f8800101be0 2024-09-16T09:17:20.551 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.548+0000 7f87f7fff640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f87ec07ae10 0x7f87ec07d2d0 secure :-1 s=READY pgs=3953 cs=0 l=1 rev1=1 crypto rx=0x7f8800110d60 tx=0x7f87e800a2d0 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:17:20.553 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.552+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f87f0069a90 con 0x7f8800101be0 2024-09-16T09:17:20.730 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:20.729+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} v 0) -- 0x7f88001097b0 con 0x7f8800101be0 2024-09-16T09:17:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:20 smithi114 ceph-mon[33315]: pgmap v1233: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:17:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:20 smithi114 ceph-mon[33315]: osdmap e760: 8 total, 8 up, 8 in 2024-09-16T09:17:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:20 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2124482270' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:20.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:20 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:20.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[29803]: pgmap v1233: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:20.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:17:20.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[29803]: osdmap e760: 8 total, 8 up, 8 in 2024-09-16T09:17:20.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2124482270' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:20.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:20.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[40056]: pgmap v1233: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:17:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[40056]: osdmap e760: 8 total, 8 up, 8 in 2024-09-16T09:17:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2124482270' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:20.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:20 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"} : dispatch 2024-09-16T09:17:21.454 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:21.453+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 7 ==== osd_map(762..762 src has 1..762) ==== 628+0+0 (secure 0 0 0) 0x7f87f00d6190 con 0x7f8800101be0 2024-09-16T09:17:21.454 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:21.453+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=763}) -- 0x7f87ec087ce0 con 0x7f8800101be0 2024-09-16T09:17:21.458 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:21.457+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v762) ==== 221+0+0 (secure 0 0 0) 0x7f87f006e940 con 0x7f8800101be0 2024-09-16T09:17:21.567 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:21.566+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} v 0) -- 0x7f8800109a00 con 0x7f8800101be0 2024-09-16T09:17:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:17:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:21 smithi114 ceph-mon[33315]: osdmap e761: 8 total, 8 up, 8 in 2024-09-16T09:17:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:21 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/539471565' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:21.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:21 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:17:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:21 smithi111 ceph-mon[40056]: osdmap e761: 8 total, 8 up, 8 in 2024-09-16T09:17:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:21 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/539471565' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:21.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:21 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:21.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_bytes", "val": "0"}]': finished 2024-09-16T09:17:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:21 smithi111 ceph-mon[29803]: osdmap e761: 8 total, 8 up, 8 in 2024-09-16T09:17:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:21 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/539471565' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:21.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:21 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:22.456 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.455+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 9 ==== osd_map(763..763 src has 1..763) ==== 628+0+0 (secure 0 0 0) 0x7f87f009c6c0 con 0x7f8800101be0 2024-09-16T09:17:22.456 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.455+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=764}) -- 0x7f87ec0882e0 con 0x7f8800101be0 2024-09-16T09:17:22.460 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.459+0000 7f87f77fe640 1 -- 172.21.15.111:0/539471565 <== mon.2 v2:172.21.15.111:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c v763) ==== 221+0+0 (secure 0 0 0) 0x7f87f008c220 con 0x7f8800101be0 2024-09-16T09:17:22.460 INFO:tasks.workunit.client.0.smithi111.stderr:set-quota max_objects = 0 for pool aa242ffd-5c69-42f2-92b2-30d0602f301c 2024-09-16T09:17:22.464 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f87ec07ae10 msgr2=0x7f87ec07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:22.464 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f87ec07ae10 0x7f87ec07d2d0 secure :-1 s=READY pgs=3953 cs=0 l=1 rev1=1 crypto rx=0x7f8800110d60 tx=0x7f87e800a2d0 comp rx=0 tx=0).stop 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800101be0 msgr2=0x7f880010f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800101be0 0x7f880010f270 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f87f000ea60 tx=0x7f87f000ee60 comp rx=0 tx=0).stop 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 shutdown_connections 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f87ec07ae10 0x7f87ec07d2d0 unknown :-1 s=CLOSED pgs=3953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8800107f60 0x7f880010fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88001075e0 0x7f880010f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 --2- 172.21.15.111:0/539471565 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8800101be0 0x7f880010f270 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 >> 172.21.15.111:0/539471565 conn(0x7f88000fd640 msgr2=0x7f88000fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 shutdown_connections 2024-09-16T09:17:22.465 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:22.463+0000 7f8804a9b640 1 -- 172.21.15.111:0/539471565 wait complete. 2024-09-16T09:17:22.477 INFO:tasks.workunit.client.0.smithi111.stderr:+ sleep 30 2024-09-16T09:17:22.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:17:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:17:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:22 smithi114 ceph-mon[33315]: pgmap v1236: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:17:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:17:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:22 smithi114 ceph-mon[33315]: osdmap e762: 8 total, 8 up, 8 in 2024-09-16T09:17:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:22 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/539471565' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:22.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:22 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[40056]: pgmap v1236: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:17:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:17:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[40056]: osdmap e762: 8 total, 8 up, 8 in 2024-09-16T09:17:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/539471565' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:22.926 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:22.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[29803]: pgmap v1236: 188 pgs: 188 active+clean; 884 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-16T09:17:22.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:17:22.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[29803]: osdmap e762: 8 total, 8 up, 8 in 2024-09-16T09:17:22.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/539471565' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:22.927 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:22 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"} : dispatch 2024-09-16T09:17:23.470 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:23 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:17:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:17:23.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:23 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:17:23.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:23 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:23.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:23 smithi114 ceph-mon[33315]: osdmap e763: 8 total, 8 up, 8 in 2024-09-16T09:17:23.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:23 smithi114 ceph-mon[33315]: pgmap v1239: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.2 KiB/s wr, 1 op/s 2024-09-16T09:17:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:23 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:17:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:23 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:23 smithi111 ceph-mon[29803]: osdmap e763: 8 total, 8 up, 8 in 2024-09-16T09:17:23.926 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:23 smithi111 ceph-mon[29803]: pgmap v1239: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.2 KiB/s wr, 1 op/s 2024-09-16T09:17:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:23 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "field": "max_objects", "val": "0"}]': finished 2024-09-16T09:17:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:23 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:23 smithi111 ceph-mon[40056]: osdmap e763: 8 total, 8 up, 8 in 2024-09-16T09:17:23.927 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:23 smithi111 ceph-mon[40056]: pgmap v1239: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.2 KiB/s wr, 1 op/s 2024-09-16T09:17:26.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:25 smithi114 ceph-mon[33315]: pgmap v1240: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 956 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-09-16T09:17:26.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:25 smithi111 ceph-mon[40056]: pgmap v1240: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 956 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-09-16T09:17:26.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:25 smithi111 ceph-mon[29803]: pgmap v1240: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 956 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-09-16T09:17:28.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:27 smithi114 ceph-mon[33315]: pgmap v1241: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-09-16T09:17:28.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:27 smithi111 ceph-mon[29803]: pgmap v1241: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-09-16T09:17:28.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:27 smithi111 ceph-mon[40056]: pgmap v1241: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-09-16T09:17:30.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:29 smithi114 ceph-mon[33315]: pgmap v1242: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-16T09:17:30.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:29 smithi111 ceph-mon[29803]: pgmap v1242: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-16T09:17:30.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:29 smithi111 ceph-mon[40056]: pgmap v1242: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-16T09:17:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:31 smithi114 ceph-mon[33315]: pgmap v1243: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2024-09-16T09:17:32.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:31 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:31 smithi111 ceph-mon[29803]: pgmap v1243: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2024-09-16T09:17:32.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:31 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:32.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:31 smithi111 ceph-mon[40056]: pgmap v1243: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2024-09-16T09:17:32.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:31 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:32.914 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:17:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:17:33.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:33 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:17:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:17:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:33 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:34.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:33 smithi114 ceph-mon[33315]: pgmap v1244: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-09-16T09:17:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:33 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:34.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:33 smithi111 ceph-mon[40056]: pgmap v1244: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-09-16T09:17:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:33 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:34.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:33 smithi111 ceph-mon[29803]: pgmap v1244: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-09-16T09:17:36.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:35 smithi114 ceph-mon[33315]: pgmap v1245: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:36.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:35 smithi111 ceph-mon[29803]: pgmap v1245: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:36.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:35 smithi111 ceph-mon[40056]: pgmap v1245: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:38.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:37 smithi114 ceph-mon[33315]: pgmap v1246: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:38.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:37 smithi111 ceph-mon[29803]: pgmap v1246: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:38.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:37 smithi111 ceph-mon[40056]: pgmap v1246: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:40.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:39 smithi114 ceph-mon[33315]: pgmap v1247: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:40.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:39 smithi111 ceph-mon[29803]: pgmap v1247: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:40.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:39 smithi111 ceph-mon[40056]: pgmap v1247: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:42.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:41 smithi114 ceph-mon[33315]: pgmap v1248: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:42.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:41 smithi111 ceph-mon[29803]: pgmap v1248: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:42.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:41 smithi111 ceph-mon[40056]: pgmap v1248: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:42.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:17:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:17:43.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:43 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:17:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:17:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:43 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:44.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:43 smithi114 ceph-mon[33315]: pgmap v1249: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:43 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:44.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:43 smithi111 ceph-mon[40056]: pgmap v1249: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:44.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:43 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:44.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:43 smithi111 ceph-mon[29803]: pgmap v1249: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:46.163 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:45 smithi114 ceph-mon[33315]: pgmap v1250: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:46.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:45 smithi111 ceph-mon[29803]: pgmap v1250: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:46.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:45 smithi111 ceph-mon[40056]: pgmap v1250: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:47.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:46 smithi114 ceph-mon[33315]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:47.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:46 smithi111 ceph-mon[29803]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:47.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:46 smithi111 ceph-mon[40056]: from='mgr.14607 172.21.15.111:0/3873194794' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-16T09:17:48.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:47 smithi114 ceph-mon[33315]: pgmap v1251: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:48.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:47 smithi111 ceph-mon[29803]: pgmap v1251: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:48.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:47 smithi111 ceph-mon[40056]: pgmap v1251: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:50.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:49 smithi114 ceph-mon[33315]: pgmap v1252: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:50.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:49 smithi111 ceph-mon[29803]: pgmap v1252: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:50.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:49 smithi111 ceph-mon[40056]: pgmap v1252: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:52.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:51 smithi114 ceph-mon[33315]: pgmap v1253: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:52.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:51 smithi111 ceph-mon[29803]: pgmap v1253: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:52.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:51 smithi111 ceph-mon[40056]: pgmap v1253: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-16T09:17:52.478 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool delete aa242ffd-5c69-42f2-92b2-30d0602f301c aa242ffd-5c69-42f2-92b2-30d0602f301c --yes-i-really-really-mean-it 2024-09-16T09:17:52.574 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.566+0000 7f9abaecd640 1 -- 172.21.15.111:0/2170928303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4102cf0 msgr2=0x7f9ab41030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:52.574 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.566+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2170928303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4102cf0 0x7f9ab41030d0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009900 tx=0x7f9aa402a410 comp rx=0 tx=0).stop 2024-09-16T09:17:52.574 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.566+0000 7f9abaecd640 1 -- 172.21.15.111:0/2170928303 shutdown_connections 2024-09-16T09:17:52.574 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.566+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2170928303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab4109070 0x7f9ab410f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:52.575 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.566+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2170928303 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ab41086f0 0x7f9ab4108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:52.575 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.566+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2170928303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4102cf0 0x7f9ab41030d0 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:52.575 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.566+0000 7f9abaecd640 1 -- 172.21.15.111:0/2170928303 >> 172.21.15.111:0/2170928303 conn(0x7f9ab40fe730 msgr2=0x7f9ab4100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:17:52.575 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.566+0000 7f9abaecd640 1 -- 172.21.15.111:0/2170928303 shutdown_connections 2024-09-16T09:17:52.575 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.566+0000 7f9abaecd640 1 -- 172.21.15.111:0/2170928303 wait complete. 2024-09-16T09:17:52.576 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9abaecd640 1 Processor -- start 2024-09-16T09:17:52.576 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9abaecd640 1 -- start start 2024-09-16T09:17:52.576 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9abaecd640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ab4102cf0 0x7f9ab4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:52.576 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9abaecd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab41086f0 0x7f9ab4198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:52.576 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9abaecd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4109070 0x7f9ab419e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:52.577 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9abaecd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab41121c0 con 0x7f9ab4109070 2024-09-16T09:17:52.577 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9abaecd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab4112040 con 0x7f9ab41086f0 2024-09-16T09:17:52.577 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9abaecd640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab4112340 con 0x7f9ab4102cf0 2024-09-16T09:17:52.577 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9ab8c69640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ab4102cf0 0x7f9ab4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:52.577 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9ab946a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4109070 0x7f9ab419e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.567+0000 7f9ab8c69640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ab4102cf0 0x7f9ab4198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:53344/0 (socket says 172.21.15.111:53344) 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab8c69640 1 -- 172.21.15.111:0/2752056696 learned_addr learned my addr 172.21.15.111:0/2752056696 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab946a640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4109070 0x7f9ab419e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53382/0 (socket says 172.21.15.111:53382) 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab3fff640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab41086f0 0x7f9ab4198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab8c69640 1 -- 172.21.15.111:0/2752056696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab41086f0 msgr2=0x7f9ab4198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab8c69640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab41086f0 0x7f9ab4198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab8c69640 1 -- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4109070 msgr2=0x7f9ab419e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab8c69640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4109070 0x7f9ab419e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab8c69640 1 -- 172.21.15.111:0/2752056696 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab419e800 con 0x7f9ab4102cf0 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab3fff640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab41086f0 0x7f9ab4198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T09:17:52.578 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab946a640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4109070 0x7f9ab419e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab8c69640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ab4102cf0 0x7f9ab4198760 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4005940 tx=0x7f9aa4032b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa403f030 con 0x7f9ab4102cf0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f9aa4003a40 con 0x7f9ab4102cf0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab419ea90 con 0x7f9ab4102cf0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4035070 con 0x7f9ab4102cf0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.568+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab41a51b0 con 0x7f9ab4102cf0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.570+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab41030d0 con 0x7f9ab4102cf0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.570+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f9aa4042030 con 0x7f9ab4102cf0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.571+0000 7f9ab1ffb640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f9aa007ae10 0x7f9aa007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.571+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(763..763 src has 1..763) ==== 9451+0+0 (secure 0 0 0) 0x7f9aa40bf0c0 con 0x7f9ab4102cf0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.571+0000 7f9ab3fff640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f9aa007ae10 0x7f9aa007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.571+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=764}) -- 0x7f9aa0087130 con 0x7f9ab4102cf0 2024-09-16T09:17:52.579 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.571+0000 7f9ab3fff640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f9aa007ae10 0x7f9aa007d2d0 secure :-1 s=READY pgs=3954 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4102010 tx=0x7f9a9c009500 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:17:52.580 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.575+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f9aa4089bb0 con 0x7f9ab4102cf0 2024-09-16T09:17:52.752 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.750+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9ab410ae50 con 0x7f9ab4102cf0 2024-09-16T09:17:52.891 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:17:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug there is no tcmu-runner data available 2024-09-16T09:17:52.896 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.894+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 <== mon.2 v2:172.21.15.111:3301/0 7 ==== osd_map(764..764 src has 1..764) ==== 296+0+0 (secure 0 0 0) 0x7f9aa40f6190 con 0x7f9ab4102cf0 2024-09-16T09:17:52.896 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.894+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=765}) -- 0x7f9aa0087a00 con 0x7f9ab4102cf0 2024-09-16T09:17:52.899 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:52.897+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 <== mon.2 v2:172.21.15.111:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true}]=0 pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' removed v764) ==== 248+0+0 (secure 0 0 0) 0x7f9aa408ea60 con 0x7f9ab4102cf0 2024-09-16T09:17:53.009 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.006+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9ab410a8c0 con 0x7f9ab4102cf0 2024-09-16T09:17:53.010 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.007+0000 7f9ab1ffb640 1 -- 172.21.15.111:0/2752056696 <== mon.2 v2:172.21.15.111:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true}]=0 pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' does not exist v764) ==== 255+0+0 (secure 0 0 0) 0x7f9aa408c080 con 0x7f9ab4102cf0 2024-09-16T09:17:53.010 INFO:tasks.workunit.client.0.smithi111.stderr:pool 'aa242ffd-5c69-42f2-92b2-30d0602f301c' does not exist 2024-09-16T09:17:53.012 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f9aa007ae10 msgr2=0x7f9aa007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:53.013 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f9aa007ae10 0x7f9aa007d2d0 secure :-1 s=READY pgs=3954 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4102010 tx=0x7f9a9c009500 comp rx=0 tx=0).stop 2024-09-16T09:17:53.013 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ab4102cf0 msgr2=0x7f9ab4198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:53.013 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ab4102cf0 0x7f9ab4198760 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4005940 tx=0x7f9aa4032b70 comp rx=0 tx=0).stop 2024-09-16T09:17:53.013 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 shutdown_connections 2024-09-16T09:17:53.013 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f9aa007ae10 0x7f9aa007d2d0 unknown :-1 s=CLOSED pgs=3954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.013 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab4109070 0x7f9ab419e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.014 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab41086f0 0x7f9ab4198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.014 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 --2- 172.21.15.111:0/2752056696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ab4102cf0 0x7f9ab4198760 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.014 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.009+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 >> 172.21.15.111:0/2752056696 conn(0x7f9ab40fe730 msgr2=0x7f9ab4100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:17:53.014 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.010+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 shutdown_connections 2024-09-16T09:17:53.014 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.010+0000 7f9abaecd640 1 -- 172.21.15.111:0/2752056696 wait complete. 2024-09-16T09:17:53.021 INFO:tasks.workunit.client.0.smithi111.stderr:+ ceph osd pool delete 4b166c41-77f3-458e-9e25-c5fe356364a4 4b166c41-77f3-458e-9e25-c5fe356364a4 --yes-i-really-really-mean-it 2024-09-16T09:17:53.108 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.102+0000 7f28cd382640 1 -- 172.21.15.111:0/2429104393 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c8109160 msgr2=0x7f28c810f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:53.108 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.102+0000 7f28cd382640 1 --2- 172.21.15.111:0/2429104393 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c8109160 0x7f28c810f890 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009be0 tx=0x7f28c002a6c0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.102+0000 7f28cd382640 1 -- 172.21.15.111:0/2429104393 shutdown_connections 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.102+0000 7f28cd382640 1 --2- 172.21.15.111:0/2429104393 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c8109160 0x7f28c810f890 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.102+0000 7f28cd382640 1 --2- 172.21.15.111:0/2429104393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28c81087e0 0x7f28c8108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.102+0000 7f28cd382640 1 --2- 172.21.15.111:0/2429104393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c81016f0 0x7f28c8101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.102+0000 7f28cd382640 1 -- 172.21.15.111:0/2429104393 >> 172.21.15.111:0/2429104393 conn(0x7f28c8077fe0 msgr2=0x7f28c80783f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.102+0000 7f28cd382640 1 -- 172.21.15.111:0/2429104393 shutdown_connections 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.102+0000 7f28cd382640 1 -- 172.21.15.111:0/2429104393 wait complete. 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28cd382640 1 Processor -- start 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28cd382640 1 -- start start 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28cd382640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c81016f0 0x7f28c819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:53.109 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28cd382640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28c81087e0 0x7f28c819ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28cd382640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c8109160 0x7f28c81a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28cd382640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f28c8112220 con 0x7f28c8109160 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28cd382640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81120a0 con 0x7f28c81087e0 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28cd382640 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 -- 0x7f28c81123a0 con 0x7f28c81016f0 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28c6ffd640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c81016f0 0x7f28c819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28c6ffd640 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c81016f0 0x7f28c819a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:53360/0 (socket says 172.21.15.111:53360) 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28c6ffd640 1 -- 172.21.15.111:0/4213539856 learned_addr learned my addr 172.21.15.111:0/4213539856 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28c77fe640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c8109160 0x7f28c81a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28bffff640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28c81087e0 0x7f28c819ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28c6ffd640 1 -- 172.21.15.111:0/4213539856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28c81087e0 msgr2=0x7f28c819ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28c6ffd640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28c81087e0 0x7f28c819ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28c6ffd640 1 -- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c8109160 msgr2=0x7f28c81a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:53.110 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28c6ffd640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c8109160 0x7f28c81a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.103+0000 7f28c6ffd640 1 -- 172.21.15.111:0/4213539856 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c81a09a0 con 0x7f28c81016f0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.104+0000 7f28c6ffd640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c81016f0 0x7f28c819a900 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f28b800e9c0 tx=0x7f28b800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.104+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b800cde0 con 0x7f28c81016f0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.104+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(23 keys) ==== 1003+0+0 (secure 0 0 0) 0x7f28b8004470 con 0x7f28c81016f0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.104+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c81a0c90 con 0x7f28c81016f0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.104+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b8003e50 con 0x7f28c81016f0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.104+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c81a7350 con 0x7f28c81016f0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.105+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c8101ad0 con 0x7f28c81016f0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.105+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 22) ==== 103138+0+0 (secure 0 0 0) 0x7f28b8004000 con 0x7f28c81016f0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.106+0000 7f28c4ff9640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f28b407afb0 0x7f28b407d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.106+0000 7f28bffff640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f28b407afb0 0x7f28b407d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.106+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(764..764 src has 1..764) ==== 9063+0+0 (secure 0 0 0) 0x7f28b809ee90 con 0x7f28c81016f0 2024-09-16T09:17:53.111 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.106+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=765}) -- 0x7f28b40869b0 con 0x7f28c81016f0 2024-09-16T09:17:53.112 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.106+0000 7f28bffff640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f28b407afb0 0x7f28b407d470 secure :-1 s=READY pgs=3955 cs=0 l=1 rev1=1 crypto rx=0x7f28b0005fd0 tx=0x7f28b0005850 comp rx=0 tx=0).ready entity=mgr.14607 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T09:17:53.112 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.111+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196987 (secure 0 0 0) 0x7f28b8069a80 con 0x7f28c81016f0 2024-09-16T09:17:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:52 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2752056696' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:53.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:52 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:53.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:52 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2752056696' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:53.176 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:52 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:52 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2752056696' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:53.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:52 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:53.294 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.292+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} v 0) -- 0x7f28c81a7610 con 0x7f28c81016f0 2024-09-16T09:17:53.676 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: ::ffff:172.21.15.114 - - [16/Sep/2024:09:17:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-16T09:17:53.897 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.895+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 <== mon.2 v2:172.21.15.111:3301/0 7 ==== osd_map(765..765 src has 1..765) ==== 296+0+0 (secure 0 0 0) 0x7f28b80035f0 con 0x7f28c81016f0 2024-09-16T09:17:53.898 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.895+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=766}) -- 0x7f28b4087240 con 0x7f28c81016f0 2024-09-16T09:17:53.901 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:53.899+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 <== mon.2 v2:172.21.15.111:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true}]=0 pool '4b166c41-77f3-458e-9e25-c5fe356364a4' removed v765) ==== 248+0+0 (secure 0 0 0) 0x7f28b806e930 con 0x7f28c81016f0 2024-09-16T09:17:54.012 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.010+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} v 0) -- 0x7f28c81a7a60 con 0x7f28c81016f0 2024-09-16T09:17:54.013 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.011+0000 7f28c4ff9640 1 -- 172.21.15.111:0/4213539856 <== mon.2 v2:172.21.15.111:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true}]=0 pool '4b166c41-77f3-458e-9e25-c5fe356364a4' does not exist v765) ==== 255+0+0 (secure 0 0 0) 0x7f28b809c730 con 0x7f28c81016f0 2024-09-16T09:17:54.013 INFO:tasks.workunit.client.0.smithi111.stderr:pool '4b166c41-77f3-458e-9e25-c5fe356364a4' does not exist 2024-09-16T09:17:54.015 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f28b407afb0 msgr2=0x7f28b407d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:54.015 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f28b407afb0 0x7f28b407d470 secure :-1 s=READY pgs=3955 cs=0 l=1 rev1=1 crypto rx=0x7f28b0005fd0 tx=0x7f28b0005850 comp rx=0 tx=0).stop 2024-09-16T09:17:54.015 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c81016f0 msgr2=0x7f28c819a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T09:17:54.015 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c81016f0 0x7f28c819a900 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f28b800e9c0 tx=0x7f28b800ee80 comp rx=0 tx=0).stop 2024-09-16T09:17:54.016 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 shutdown_connections 2024-09-16T09:17:54.016 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:6800/1888355313,v1:172.21.15.111:6801/1888355313] conn(0x7f28b407afb0 0x7f28b407d470 unknown :-1 s=CLOSED pgs=3955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:54.016 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28c8109160 0x7f28c81a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:54.016 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28c81087e0 0x7f28c819ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:54.016 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 --2- 172.21.15.111:0/4213539856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28c81016f0 0x7f28c819a900 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T09:17:54.016 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 >> 172.21.15.111:0/4213539856 conn(0x7f28c8077fe0 msgr2=0x7f28c8100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T09:17:54.016 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 shutdown_connections 2024-09-16T09:17:54.016 INFO:tasks.workunit.client.0.smithi111.stderr:2024-09-16T09:17:54.013+0000 7f28cd382640 1 -- 172.21.15.111:0/4213539856 wait complete. 2024-09-16T09:17:54.024 INFO:tasks.workunit.client.0.smithi111.stdout:OK 2024-09-16T09:17:54.025 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo OK 2024-09-16T09:17:54.026 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T09:17:54.026 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T09:17:54.093 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-09-16T09:17:54.094 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T09:17:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:53 smithi114 ceph-mon[33315]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:53 smithi114 ceph-mon[33315]: pgmap v1254: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T09:17:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:53 smithi114 ceph-mon[33315]: osdmap e764: 8 total, 8 up, 8 in 2024-09-16T09:17:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/2752056696' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:53 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4213539856' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:53 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[40056]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:54.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[40056]: pgmap v1254: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:54.176 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T09:17:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[40056]: osdmap e764: 8 total, 8 up, 8 in 2024-09-16T09:17:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/2752056696' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4213539856' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.177 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[29803]: from='client.24436 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-16T09:17:54.177 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[29803]: pgmap v1254: 188 pgs: 188 active+clean; 897 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-16T09:17:54.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T09:17:54.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[29803]: osdmap e764: 8 total, 8 up, 8 in 2024-09-16T09:17:54.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/2752056696' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "pool2": "aa242ffd-5c69-42f2-92b2-30d0602f301c", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4213539856' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.178 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:53 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:54.410 DEBUG:teuthology.parallel:result is None 2024-09-16T09:17:54.411 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T09:17:54.476 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T09:17:54.477 DEBUG:teuthology.orchestra.run.smithi111:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T09:17:54.552 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T09:17:54.552 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-16T09:17:54.562 INFO:tasks.cephadm:Teardown begin 2024-09-16T09:17:54.562 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T09:17:54.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T09:17:54.650 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-16T09:17:54.650 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T09:17:54.674 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T09:17:54.708 INFO:tasks.cephadm:Stopping all daemons... 2024-09-16T09:17:54.709 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-16T09:17:54.709 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.a 2024-09-16T09:17:55.060 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[40056]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:17:55.060 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T09:17:55.060 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[40056]: osdmap e765: 8 total, 8 up, 8 in 2024-09-16T09:17:55.060 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[40056]: from='client.? 172.21.15.111:0/4213539856' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:55.061 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[40056]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:55.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:54 smithi111 systemd[1]: Stopping Ceph mon.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:17:55.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[29803]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:17:55.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T09:17:55.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[29803]: osdmap e765: 8 total, 8 up, 8 in 2024-09-16T09:17:55.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[29803]: from='client.? 172.21.15.111:0/4213539856' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:55.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:54 smithi111 ceph-mon[29803]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:54 smithi114 ceph-mon[33315]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-16T09:17:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:54 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T09:17:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:54 smithi114 ceph-mon[33315]: osdmap e765: 8 total, 8 up, 8 in 2024-09-16T09:17:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:54 smithi114 ceph-mon[33315]: from='client.? 172.21.15.111:0/4213539856' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:55.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:54 smithi114 ceph-mon[33315]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool delete", "pool": "4b166c41-77f3-458e-9e25-c5fe356364a4", "pool2": "4b166c41-77f3-458e-9e25-c5fe356364a4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-16T09:17:55.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a[29780]: 2024-09-16T09:17:55.058+0000 7f72e7f11640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:17:55.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a[29780]: 2024-09-16T09:17:55.058+0000 7f72e7f11640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-16T09:17:55.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:55 smithi111 podman[147475]: 2024-09-16 09:17:55.100038259 +0000 UTC m=+0.201533521 container died 676e47c984e2ea0d1182324f64bd7114a8677a1ebc2178b05fd635f520f22dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-16T09:17:55.394 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:17:55] ENGINE Bus STOPPING 2024-09-16T09:17:55.394 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:17:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-16T09:17:55.394 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:17:55] ENGINE Bus STOPPED 2024-09-16T09:17:55.394 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:17:55] ENGINE Bus STARTING 2024-09-16T09:17:55.394 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:17:55] ENGINE Serving on http://:::9283 2024-09-16T09:17:55.394 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:55 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y[30054]: [16/Sep/2024:09:17:55] ENGINE Bus STARTED 2024-09-16T09:17:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:55 smithi111 podman[147475]: 2024-09-16 09:17:55.575104191 +0000 UTC m=+0.676599491 container cleanup 676e47c984e2ea0d1182324f64bd7114a8677a1ebc2178b05fd635f520f22dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T09:17:55.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Sep 16 09:17:55 smithi111 bash[147475]: ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-a 2024-09-16T09:17:56.006 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.a.service' 2024-09-16T09:17:56.043 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:17:56.043 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-16T09:17:56.044 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-16T09:17:56.044 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.c 2024-09-16T09:17:56.408 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:56 smithi111 systemd[1]: Stopping Ceph mon.c for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:17:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:56 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-c[40033]: 2024-09-16T09:17:56.406+0000 7f1d456d2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:17:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:56 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-c[40033]: 2024-09-16T09:17:56.406+0000 7f1d456d2640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-16T09:17:56.676 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:56 smithi111 podman[147618]: 2024-09-16 09:17:56.509682443 +0000 UTC m=+0.250590327 container died 86f0e47425e123e3f3dd2975809f219802435986def775d475f6fee739970295 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-c, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-16T09:17:57.116 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:56 smithi111 podman[147618]: 2024-09-16 09:17:56.880461098 +0000 UTC m=+0.621368983 container cleanup 86f0e47425e123e3f3dd2975809f219802435986def775d475f6fee739970295 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-16T09:17:57.116 INFO:journalctl@ceph.mon.c.smithi111.stdout:Sep 16 09:17:56 smithi111 bash[147618]: ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-c 2024-09-16T09:17:57.262 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.c.service' 2024-09-16T09:17:57.296 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:17:57.296 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-16T09:17:57.296 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-16T09:17:57.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.b 2024-09-16T09:17:57.609 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:57 smithi114 systemd[1]: Stopping Ceph mon.b for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:17:57.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:57 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-b[33285]: 2024-09-16T09:17:57.607+0000 7f1db26de640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:17:57.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:57 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-b[33285]: 2024-09-16T09:17:57.607+0000 7f1db26de640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-16T09:17:57.914 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:57 smithi114 podman[67759]: 2024-09-16 09:17:57.659808738 +0000 UTC m=+0.164468493 container died 4c26029b2e1217e676f80bae947be0b0cdaee9666704adcd2208563666737271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T09:17:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:58 smithi114 podman[67759]: 2024-09-16 09:17:58.122519582 +0000 UTC m=+0.627179333 container cleanup 4c26029b2e1217e676f80bae947be0b0cdaee9666704adcd2208563666737271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-b, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-16T09:17:58.414 INFO:journalctl@ceph.mon.b.smithi114.stdout:Sep 16 09:17:58 smithi114 bash[67759]: ceph-f774c998-7408-11ef-bceb-c7b262605968-mon-b 2024-09-16T09:17:58.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mon.b.service' 2024-09-16T09:17:58.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:17:58.656 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-16T09:17:58.656 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-16T09:17:58.656 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.y 2024-09-16T09:17:59.176 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:58 smithi111 systemd[1]: Stopping Ceph mgr.y for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:17:59.612 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:59 smithi111 podman[147750]: 2024-09-16 09:17:59.31870535 +0000 UTC m=+0.472068096 container died 5d7785ae384e4632538cd532d952d65838412b5b7fb6cbe112399a0851a4db5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T09:17:59.921 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:59 smithi111 podman[147750]: 2024-09-16 09:17:59.719039668 +0000 UTC m=+0.872402400 container cleanup 5d7785ae384e4632538cd532d952d65838412b5b7fb6cbe112399a0851a4db5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8293d73f8690540e843a81caec373f9cc29cf705, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-16T09:17:59.921 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Sep 16 09:17:59 smithi111 bash[147750]: ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-y 2024-09-16T09:18:00.142 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-16T09:18:00.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:18:00.177 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-16T09:18:00.177 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-16T09:18:00.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.x 2024-09-16T09:18:00.501 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 09:18:00 smithi114 systemd[1]: Stopping Ceph mgr.x for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:18:00.782 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 09:18:00 smithi114 podman[67894]: 2024-09-16 09:18:00.50027148 +0000 UTC m=+0.132334555 container died f108ac6811d146ca0a8c3bc199c0e68d6dbeedd2bfe2806af34558ad7b6e05cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-16T09:18:01.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 09:18:00 smithi114 podman[67894]: 2024-09-16 09:18:00.885557762 +0000 UTC m=+0.517620837 container cleanup f108ac6811d146ca0a8c3bc199c0e68d6dbeedd2bfe2806af34558ad7b6e05cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T09:18:01.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 16 09:18:00 smithi114 bash[67894]: ceph-f774c998-7408-11ef-bceb-c7b262605968-mgr-x 2024-09-16T09:18:01.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-16T09:18:01.359 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:18:01.359 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-16T09:18:01.360 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-16T09:18:01.360 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.0 2024-09-16T09:18:01.667 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:01 smithi111 systemd[1]: Stopping Ceph osd.0 for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:18:01.926 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:01 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0[43720]: 2024-09-16T09:18:01.666+0000 7fd99364a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:18:01.926 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:01 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0[43720]: 2024-09-16T09:18:01.666+0000 7fd99364a640 -1 osd.0 765 *** Got signal Terminated *** 2024-09-16T09:18:01.926 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:01 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0[43720]: 2024-09-16T09:18:01.666+0000 7fd99364a640 -1 osd.0 765 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T09:18:07.425 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:07 smithi111 podman[147880]: 2024-09-16 09:18:07.126107529 +0000 UTC m=+5.569216757 container died 08ea0a9547e02586d3991aac7a8d34d69e7d3f6eeacc3d4577fde6fb77228818 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T09:18:07.676 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:07 smithi111 podman[147880]: 2024-09-16 09:18:07.532032568 +0000 UTC m=+5.975141793 container cleanup 08ea0a9547e02586d3991aac7a8d34d69e7d3f6eeacc3d4577fde6fb77228818 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-16T09:18:07.676 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:07 smithi111 bash[147880]: ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0 2024-09-16T09:18:08.144 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:07 smithi111 podman[147901]: 2024-09-16 09:18:07.725662143 +0000 UTC m=+0.598874578 container remove 08ea0a9547e02586d3991aac7a8d34d69e7d3f6eeacc3d4577fde6fb77228818 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2) 2024-09-16T09:18:08.405 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:08 smithi111 podman[147994]: 2024-09-16 09:18:08.046470801 +0000 UTC m=+0.015795773 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T09:18:08.405 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:08 smithi111 podman[147994]: 2024-09-16 09:18:08.18499289 +0000 UTC m=+0.154317850 container create 1f0c8cd32dd72cfab94bd7361a7ccf730b35e5ecf270920b616479a81d97cec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T09:18:08.676 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:08 smithi111 podman[147994]: 2024-09-16 09:18:08.404313908 +0000 UTC m=+0.373638868 container init 1f0c8cd32dd72cfab94bd7361a7ccf730b35e5ecf270920b616479a81d97cec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-16T09:18:08.677 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:08 smithi111 podman[147994]: 2024-09-16 09:18:08.409219146 +0000 UTC m=+0.378544106 container start 1f0c8cd32dd72cfab94bd7361a7ccf730b35e5ecf270920b616479a81d97cec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T09:18:08.677 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:08 smithi111 podman[147994]: 2024-09-16 09:18:08.496009377 +0000 UTC m=+0.465334344 container attach 1f0c8cd32dd72cfab94bd7361a7ccf730b35e5ecf270920b616479a81d97cec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-16T09:18:08.677 INFO:journalctl@ceph.osd.0.smithi111.stdout:Sep 16 09:18:08 smithi111 podman[147994]: 2024-09-16 09:18:08.605957414 +0000 UTC m=+0.575282387 container died 1f0c8cd32dd72cfab94bd7361a7ccf730b35e5ecf270920b616479a81d97cec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-16T09:18:09.207 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.0.service' 2024-09-16T09:18:09.251 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:18:09.251 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-16T09:18:09.251 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-16T09:18:09.251 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.1 2024-09-16T09:18:09.488 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:09 smithi111 systemd[1]: Stopping Ceph osd.1 for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:18:09.926 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:09 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1[49769]: 2024-09-16T09:18:09.620+0000 7fb57552c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:18:09.926 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:09 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1[49769]: 2024-09-16T09:18:09.620+0000 7fb57552c640 -1 osd.1 765 *** Got signal Terminated *** 2024-09-16T09:18:09.926 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:09 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1[49769]: 2024-09-16T09:18:09.620+0000 7fb57552c640 -1 osd.1 765 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T09:18:15.362 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:15 smithi111 podman[148125]: 2024-09-16 09:18:15.085889997 +0000 UTC m=+5.580830686 container died 6ad8dd93c7f7d8fb873c9273c7ac7ab78b0a8b717213d2d2335ab2d2253a3944 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-16T09:18:15.676 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:15 smithi111 podman[148125]: 2024-09-16 09:18:15.460541408 +0000 UTC m=+5.955482086 container cleanup 6ad8dd93c7f7d8fb873c9273c7ac7ab78b0a8b717213d2d2335ab2d2253a3944 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-16T09:18:15.676 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:15 smithi111 bash[148125]: ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1 2024-09-16T09:18:16.083 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:15 smithi111 podman[148157]: 2024-09-16 09:18:15.737030394 +0000 UTC m=+0.648576913 container remove 6ad8dd93c7f7d8fb873c9273c7ac7ab78b0a8b717213d2d2335ab2d2253a3944 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-16T09:18:16.342 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:16 smithi111 podman[148254]: 2024-09-16 09:18:15.985797504 +0000 UTC m=+0.015962927 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T09:18:16.342 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:16 smithi111 podman[148254]: 2024-09-16 09:18:16.087754681 +0000 UTC m=+0.117920093 container create 0eec5053759ec6ab75d0e720a30f8bc6ce372e96a46ac51b4f401f880a98a7f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-16T09:18:16.342 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:16 smithi111 podman[148254]: 2024-09-16 09:18:16.341020956 +0000 UTC m=+0.371186370 container init 0eec5053759ec6ab75d0e720a30f8bc6ce372e96a46ac51b4f401f880a98a7f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.vendor=CentOS) 2024-09-16T09:18:16.676 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:16 smithi111 podman[148254]: 2024-09-16 09:18:16.346034124 +0000 UTC m=+0.376199540 container start 0eec5053759ec6ab75d0e720a30f8bc6ce372e96a46ac51b4f401f880a98a7f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-16T09:18:16.676 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:16 smithi111 podman[148254]: 2024-09-16 09:18:16.432510283 +0000 UTC m=+0.462675705 container attach 0eec5053759ec6ab75d0e720a30f8bc6ce372e96a46ac51b4f401f880a98a7f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-16T09:18:16.676 INFO:journalctl@ceph.osd.1.smithi111.stdout:Sep 16 09:18:16 smithi111 podman[148254]: 2024-09-16 09:18:16.534578119 +0000 UTC m=+0.564743532 container died 0eec5053759ec6ab75d0e720a30f8bc6ce372e96a46ac51b4f401f880a98a7f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-16T09:18:17.134 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.1.service' 2024-09-16T09:18:17.180 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:18:17.180 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-16T09:18:17.180 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-16T09:18:17.180 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.2 2024-09-16T09:18:17.426 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:17 smithi111 systemd[1]: Stopping Ceph osd.2 for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:18:17.926 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:17 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2[55932]: 2024-09-16T09:18:17.461+0000 7f0b79d7a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:18:17.926 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:17 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2[55932]: 2024-09-16T09:18:17.461+0000 7f0b79d7a640 -1 osd.2 765 *** Got signal Terminated *** 2024-09-16T09:18:17.926 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:17 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2[55932]: 2024-09-16T09:18:17.461+0000 7f0b79d7a640 -1 osd.2 765 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T09:18:23.176 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:22 smithi111 podman[148381]: 2024-09-16 09:18:22.857562191 +0000 UTC m=+5.490639400 container died ee4c8665389071f519c375e84ec8c739429bcb368f80dd8b58f316940be5e2ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-16T09:18:23.676 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:23 smithi111 podman[148381]: 2024-09-16 09:18:23.180389205 +0000 UTC m=+5.813466416 container cleanup ee4c8665389071f519c375e84ec8c739429bcb368f80dd8b58f316940be5e2ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS) 2024-09-16T09:18:23.676 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:23 smithi111 bash[148381]: ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2 2024-09-16T09:18:23.676 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:23 smithi111 podman[148402]: 2024-09-16 09:18:23.390110268 +0000 UTC m=+0.526359086 container remove ee4c8665389071f519c375e84ec8c739429bcb368f80dd8b58f316940be5e2ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-16T09:18:24.053 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:23 smithi111 podman[148496]: 2024-09-16 09:18:23.695297187 +0000 UTC m=+0.015902639 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T09:18:24.053 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:23 smithi111 podman[148496]: 2024-09-16 09:18:23.799305023 +0000 UTC m=+0.119910513 container create 68c97b128ab247b933639c719483ef53c133224d7405a69a0b69b79f42930d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73) 2024-09-16T09:18:24.053 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:24 smithi111 podman[148496]: 2024-09-16 09:18:24.052166742 +0000 UTC m=+0.372772196 container init 68c97b128ab247b933639c719483ef53c133224d7405a69a0b69b79f42930d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T09:18:24.426 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:24 smithi111 podman[148496]: 2024-09-16 09:18:24.057225461 +0000 UTC m=+0.377830907 container start 68c97b128ab247b933639c719483ef53c133224d7405a69a0b69b79f42930d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-16T09:18:24.427 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:24 smithi111 podman[148496]: 2024-09-16 09:18:24.168758527 +0000 UTC m=+0.489363977 container attach 68c97b128ab247b933639c719483ef53c133224d7405a69a0b69b79f42930d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-16T09:18:24.427 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:24 smithi111 podman[148496]: 2024-09-16 09:18:24.243985707 +0000 UTC m=+0.564591164 container died 68c97b128ab247b933639c719483ef53c133224d7405a69a0b69b79f42930d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-16T09:18:24.780 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.2.service' 2024-09-16T09:18:24.805 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:24 smithi111 podman[148539]: 2024-09-16 09:18:24.755181812 +0000 UTC m=+0.505673966 container remove 68c97b128ab247b933639c719483ef53c133224d7405a69a0b69b79f42930d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-16T09:18:24.805 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:24 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-16T09:18:24.805 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:24 smithi111 systemd[1]: Stopped Ceph osd.2 for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:18:24.805 INFO:journalctl@ceph.osd.2.smithi111.stdout:Sep 16 09:18:24 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.2.service: Consumed 35.466s CPU time. 2024-09-16T09:18:24.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:18:24.821 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-16T09:18:24.821 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-16T09:18:24.821 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.3 2024-09-16T09:18:25.153 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:24 smithi111 systemd[1]: Stopping Ceph osd.3 for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:18:25.426 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:25 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3[62019]: 2024-09-16T09:18:25.151+0000 7f0987f26640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:18:25.426 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:25 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3[62019]: 2024-09-16T09:18:25.151+0000 7f0987f26640 -1 osd.3 765 *** Got signal Terminated *** 2024-09-16T09:18:25.426 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:25 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3[62019]: 2024-09-16T09:18:25.151+0000 7f0987f26640 -1 osd.3 765 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T09:18:27.934 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:27 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:27.662+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:29.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:28 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:28.707+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:30.160 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:29 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:29.681+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:30.413 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:30 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:30.191+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:30.426 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:30 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3[62019]: 2024-09-16T09:18:30.099+0000 7f0980533640 -1 osd.3 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:04.116757+0000 front 2024-09-16T09:18:04.116943+0000 (oldest deadline 2024-09-16T09:18:29.416891+0000) 2024-09-16T09:18:30.985 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:30 smithi111 podman[148623]: 2024-09-16 09:18:30.679602826 +0000 UTC m=+5.643187547 container died 13fda0cfa90b28f371a2ad3a9df9ae76b954857b76178ba8ce2374a132ef0822 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T09:18:31.134 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:30 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:30.699+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:31.268 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:31 smithi111 podman[148623]: 2024-09-16 09:18:31.082840489 +0000 UTC m=+6.046425207 container cleanup 13fda0cfa90b28f371a2ad3a9df9ae76b954857b76178ba8ce2374a132ef0822 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-16T09:18:31.269 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:31 smithi111 bash[148623]: ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3 2024-09-16T09:18:31.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:31.174+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:31.414 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:31.133+0000 7f8ef173f640 -1 osd.4 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:05.808228+0000 front 2024-09-16T09:18:05.808172+0000 (oldest deadline 2024-09-16T09:18:30.508127+0000) 2024-09-16T09:18:31.676 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:31 smithi111 podman[148647]: 2024-09-16 09:18:31.267631396 +0000 UTC m=+0.582624401 container remove 13fda0cfa90b28f371a2ad3a9df9ae76b954857b76178ba8ce2374a132ef0822 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73) 2024-09-16T09:18:31.738 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:18:31.463Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-09-16T09:18:31.738 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:18:31.464Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-09-16T09:18:31.738 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:18:31.467Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-09-16T09:18:31.738 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:18:31.467Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-09-16T09:18:31.738 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:18:31.468Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-09-16T09:18:31.738 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:18:31.472Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-09-16T09:18:31.738 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:18:31.472Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-09-16T09:18:31.954 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:31 smithi111 podman[148739]: 2024-09-16 09:18:31.591648405 +0000 UTC m=+0.015724760 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T09:18:31.955 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:31 smithi111 podman[148739]: 2024-09-16 09:18:31.701646396 +0000 UTC m=+0.125722746 container create cfd7c66c36a5201eb94e594f66247790fa39e93de2b4cb9839d9f3af630c25ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T09:18:31.955 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:31 smithi111 podman[148739]: 2024-09-16 09:18:31.878923169 +0000 UTC m=+0.302999526 container init cfd7c66c36a5201eb94e594f66247790fa39e93de2b4cb9839d9f3af630c25ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-09-16T09:18:31.955 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:31 smithi111 podman[148739]: 2024-09-16 09:18:31.883756831 +0000 UTC m=+0.307833183 container start cfd7c66c36a5201eb94e594f66247790fa39e93de2b4cb9839d9f3af630c25ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73) 2024-09-16T09:18:31.955 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:31 smithi111 podman[148739]: 2024-09-16 09:18:31.953820597 +0000 UTC m=+0.377896955 container attach cfd7c66c36a5201eb94e594f66247790fa39e93de2b4cb9839d9f3af630c25ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-16T09:18:32.163 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:31 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:31.736+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:32.308 INFO:journalctl@ceph.osd.3.smithi111.stdout:Sep 16 09:18:32 smithi111 podman[148739]: 2024-09-16 09:18:32.073026394 +0000 UTC m=+0.497102751 container died cfd7c66c36a5201eb94e594f66247790fa39e93de2b4cb9839d9f3af630c25ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-09-16T09:18:32.584 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.3.service' 2024-09-16T09:18:32.621 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:18:32.622 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-16T09:18:32.622 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-16T09:18:32.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.4 2024-09-16T09:18:32.651 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:32.165+0000 7f8ef173f640 -1 osd.4 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:05.808228+0000 front 2024-09-16T09:18:05.808172+0000 (oldest deadline 2024-09-16T09:18:30.508127+0000) 2024-09-16T09:18:32.651 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:32.222+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:32.914 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:32 smithi114 systemd[1]: Stopping Ceph osd.4 for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:18:32.914 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:32.740+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:33.216 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:32.914+0000 7f8ef512a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:18:33.216 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:32.914+0000 7f8ef512a640 -1 osd.4 765 *** Got signal Terminated *** 2024-09-16T09:18:33.216 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:32 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:32.914+0000 7f8ef512a640 -1 osd.4 765 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T09:18:33.216 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:33 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:33.163+0000 7f8ef173f640 -1 osd.4 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:05.808228+0000 front 2024-09-16T09:18:05.808172+0000 (oldest deadline 2024-09-16T09:18:30.508127+0000) 2024-09-16T09:18:33.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:33 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:33.340+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:33.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:33 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:33.213+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:34.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:33 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:33.765+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:34.664 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:34 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:34.198+0000 7f8ef173f640 -1 osd.4 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:05.808228+0000 front 2024-09-16T09:18:05.808172+0000 (oldest deadline 2024-09-16T09:18:30.508127+0000) 2024-09-16T09:18:34.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:34 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:34.314+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:34.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:34 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:34.247+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:35.163 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:34 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:34.720+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:35.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:35 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:35.308+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:35.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:35 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:35.234+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:35.414 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:35 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:35.162+0000 7f8ef173f640 -1 osd.4 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:05.808228+0000 front 2024-09-16T09:18:05.808172+0000 (oldest deadline 2024-09-16T09:18:30.508127+0000) 2024-09-16T09:18:36.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:35 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:35.753+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:36.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:35 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:35.753+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:36.664 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:36 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:36.166+0000 7f8ef173f640 -1 osd.4 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:05.808228+0000 front 2024-09-16T09:18:05.808172+0000 (oldest deadline 2024-09-16T09:18:30.508127+0000) 2024-09-16T09:18:36.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:36 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:36.285+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:36.665 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:36 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:36.220+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:37.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:36 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:36.797+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:37.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:36 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:36.797+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:37.426 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:18:37 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:18:37.023Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-09-16T09:18:37.426 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:18:37 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:18:37.024Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-09-16T09:18:37.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:37 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:37.215+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:37.664 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:37 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4[37735]: 2024-09-16T09:18:37.213+0000 7f8ef173f640 -1 osd.4 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:05.808228+0000 front 2024-09-16T09:18:05.808172+0000 (oldest deadline 2024-09-16T09:18:30.508127+0000) 2024-09-16T09:18:37.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:37 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:37.302+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:37.665 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:37 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:37.302+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:38.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:37 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:37.831+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:38.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:37 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:37.831+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:38.664 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:38 smithi114 podman[68041]: 2024-09-16 09:18:38.458284962 +0000 UTC m=+5.640563687 container died aee9d6f42dda55f2857ac64e7895fa3cd644163176b08bcf7310fc5be14fb959 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-16T09:18:38.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:38 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:38.241+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:38.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:38 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:38.289+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:38.665 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:38 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:38.289+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:39.073 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:38 smithi114 podman[68041]: 2024-09-16 09:18:38.829878484 +0000 UTC m=+6.012157197 container cleanup aee9d6f42dda55f2857ac64e7895fa3cd644163176b08bcf7310fc5be14fb959 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-16T09:18:39.074 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:38 smithi114 bash[68041]: ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4 2024-09-16T09:18:39.074 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:38 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:38.819+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:39.074 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:38 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:38.819+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:39.399 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:39 smithi114 podman[68061]: 2024-09-16 09:18:39.073020777 +0000 UTC m=+0.608344875 container remove aee9d6f42dda55f2857ac64e7895fa3cd644163176b08bcf7310fc5be14fb959 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T09:18:39.400 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:39 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:39.270+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:39.400 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:39 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:39.270+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:39.400 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:39 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:39.234+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:39.400 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:39 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:39.234+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:39.664 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:39 smithi114 podman[68153]: 2024-09-16 09:18:39.301730169 +0000 UTC m=+0.016149479 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T09:18:39.664 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:39 smithi114 podman[68153]: 2024-09-16 09:18:39.423885712 +0000 UTC m=+0.138305007 container create aeff194d427f999e8ca24e5067984ca76a3875052868b83a7b43d4ecd974e6b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-16T09:18:39.664 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:39 smithi114 podman[68153]: 2024-09-16 09:18:39.642915545 +0000 UTC m=+0.357334844 container init aeff194d427f999e8ca24e5067984ca76a3875052868b83a7b43d4ecd974e6b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-16T09:18:39.664 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:39 smithi114 podman[68153]: 2024-09-16 09:18:39.647896283 +0000 UTC m=+0.362315577 container start aeff194d427f999e8ca24e5067984ca76a3875052868b83a7b43d4ecd974e6b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-16T09:18:40.130 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:39 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:39.776+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:40.131 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:39 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:39.776+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:40.131 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:39 smithi114 podman[68153]: 2024-09-16 09:18:39.751152953 +0000 UTC m=+0.465572263 container attach aeff194d427f999e8ca24e5067984ca76a3875052868b83a7b43d4ecd974e6b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-16T09:18:40.131 INFO:journalctl@ceph.osd.4.smithi114.stdout:Sep 16 09:18:39 smithi114 podman[68153]: 2024-09-16 09:18:39.840368262 +0000 UTC m=+0.554787558 container died aeff194d427f999e8ca24e5067984ca76a3875052868b83a7b43d4ecd974e6b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-16T09:18:40.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:40 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:40.314+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:40.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:40 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:40.314+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:40.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:40 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:40.276+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:40.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:40 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:40.276+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:40.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.4.service' 2024-09-16T09:18:40.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:18:40.518 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-16T09:18:40.518 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-16T09:18:40.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.5 2024-09-16T09:18:40.756 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:40 smithi114 systemd[1]: Stopping Ceph osd.5 for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:18:40.756 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:40 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:40.755+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:40.756 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:40 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:40.755+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:41.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:40 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:40.859+0000 7f89a88ba640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:18:41.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:40 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:40.859+0000 7f89a88ba640 -1 osd.5 765 *** Got signal Terminated *** 2024-09-16T09:18:41.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:40 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:40.859+0000 7f89a88ba640 -1 osd.5 765 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T09:18:41.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:41.350+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:41.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:41.350+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:41.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:41.260+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:41.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:41.260+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:42.163 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:41.719+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:42.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:41 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:41.719+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:42.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:42.222+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:42.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:42.222+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:42.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:42.342+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:42.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:42.342+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:43.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:42.701+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:43.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:42 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:42.701+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:43.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:43 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:43.242+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:43.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:43 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:43.242+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:43.665 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:43 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:43.387+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:43.665 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:43 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:43.387+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:44.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:43 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:43.700+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:44.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:43 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:43.700+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:44.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:43 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:43.700+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:20.024570+0000 front 2024-09-16T09:18:20.024686+0000 (oldest deadline 2024-09-16T09:18:43.524477+0000) 2024-09-16T09:18:44.653 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:44 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:44.386+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:44.653 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:44 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:44.386+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:44.653 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:44 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:44.386+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:44.654 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:44 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:44.193+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:44.654 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:44 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:44.193+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:44.914 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:44 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:44.652+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:44.914 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:44 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:44.652+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:44.914 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:44 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:44.652+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:20.024570+0000 front 2024-09-16T09:18:20.024686+0000 (oldest deadline 2024-09-16T09:18:43.524477+0000) 2024-09-16T09:18:45.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:45 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:45.352+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:45.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:45 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:45.352+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:45.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:45 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:45.352+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:45.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:45 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:45.215+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:45.665 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:45 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:45.216+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:45.665 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:45 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:45.216+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:46.163 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:45 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:45.692+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:01.322406+0000 front 2024-09-16T09:18:01.322397+0000 (oldest deadline 2024-09-16T09:18:27.222290+0000) 2024-09-16T09:18:46.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:45 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:45.692+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:11.223706+0000 front 2024-09-16T09:18:11.223767+0000 (oldest deadline 2024-09-16T09:18:35.323607+0000) 2024-09-16T09:18:46.164 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:45 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5[43404]: 2024-09-16T09:18:45.692+0000 7f89a4ecf640 -1 osd.5 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:20.024570+0000 front 2024-09-16T09:18:20.024686+0000 (oldest deadline 2024-09-16T09:18:43.524477+0000) 2024-09-16T09:18:46.444 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:46 smithi114 podman[68282]: 2024-09-16 09:18:46.181558105 +0000 UTC m=+5.443983323 container died caece40fa7c1625419b86aae69a65d80e9d6d6a6980ebe50aba72968c14cf5b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-09-16T09:18:46.444 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:46 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:46.338+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:46.444 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:46 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:46.338+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:46.444 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:46 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:46.338+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:46.445 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:46 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:46.174+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:46.445 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:46 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:46.174+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:46.445 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:46 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:46.174+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:46.801 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:46 smithi114 podman[68282]: 2024-09-16 09:18:46.54954353 +0000 UTC m=+5.811968787 container cleanup caece40fa7c1625419b86aae69a65d80e9d6d6a6980ebe50aba72968c14cf5b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-16T09:18:46.801 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:46 smithi114 bash[68282]: ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5 2024-09-16T09:18:47.120 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:46 smithi114 podman[68304]: 2024-09-16 09:18:46.800696045 +0000 UTC m=+0.618297003 container remove caece40fa7c1625419b86aae69a65d80e9d6d6a6980ebe50aba72968c14cf5b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-16T09:18:47.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:47 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:47.339+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:47.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:47 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:47.339+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:47.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:47 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:47.339+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:47.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:47 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:47.189+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:47.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:47 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:47.189+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:47.415 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:47 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:47.189+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:47.415 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:47 smithi114 podman[68396]: 2024-09-16 09:18:47.022681198 +0000 UTC m=+0.016077776 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T09:18:47.415 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:47 smithi114 podman[68396]: 2024-09-16 09:18:47.143025794 +0000 UTC m=+0.136422370 container create 19f115e8a8a6f683f0f9bd5b831fdb011b073b6b076d9e7a8cf7331c603e1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-16T09:18:47.415 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:47 smithi114 podman[68396]: 2024-09-16 09:18:47.337049596 +0000 UTC m=+0.330446172 container init 19f115e8a8a6f683f0f9bd5b831fdb011b073b6b076d9e7a8cf7331c603e1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T09:18:47.415 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:47 smithi114 podman[68396]: 2024-09-16 09:18:47.342163779 +0000 UTC m=+0.335560356 container start 19f115e8a8a6f683f0f9bd5b831fdb011b073b6b076d9e7a8cf7331c603e1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-16T09:18:47.426 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:18:47 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:18:47.023Z caller=dispatch.go:353 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"\": dial tcp 172.21.15.114:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-09-16T09:18:47.426 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:18:47 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:18:47.024Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-09-16T09:18:47.426 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:18:47 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:18:47.025Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-09-16T09:18:47.825 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:47 smithi114 podman[68396]: 2024-09-16 09:18:47.445487946 +0000 UTC m=+0.438884524 container attach 19f115e8a8a6f683f0f9bd5b831fdb011b073b6b076d9e7a8cf7331c603e1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-16T09:18:47.825 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:47 smithi114 podman[68396]: 2024-09-16 09:18:47.530846795 +0000 UTC m=+0.524243379 container died 19f115e8a8a6f683f0f9bd5b831fdb011b073b6b076d9e7a8cf7331c603e1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-16T09:18:48.077 INFO:journalctl@ceph.osd.5.smithi114.stdout:Sep 16 09:18:48 smithi114 podman[68439]: 2024-09-16 09:18:48.065579372 +0000 UTC m=+0.528975148 container remove 19f115e8a8a6f683f0f9bd5b831fdb011b073b6b076d9e7a8cf7331c603e1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-16T09:18:48.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.5.service' 2024-09-16T09:18:48.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:18:48.122 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-16T09:18:48.122 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-16T09:18:48.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.6 2024-09-16T09:18:48.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:48 smithi114 systemd[1]: Stopping Ceph osd.6 for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:18:48.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:48 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:48.305+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:48.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:48 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:48.305+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:48.414 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:48 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:48.305+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:48.415 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:48 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:48.232+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:48.415 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:48 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:48.232+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:48.415 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:48 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:48.232+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:48.914 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:48 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:48.453+0000 7fbe9119c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:18:48.914 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:48 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:48.453+0000 7fbe9119c640 -1 osd.6 765 *** Got signal Terminated *** 2024-09-16T09:18:48.914 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:48 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:48.453+0000 7fbe9119c640 -1 osd.6 765 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T09:18:49.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:49 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:49.240+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:49.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:49 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:49.240+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:49.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:49 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:49.240+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:49.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:49 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:49.240+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:49.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:49 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:49.316+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:49.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:49 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:49.316+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:49.665 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:49 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:49.316+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:50.663 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:50 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:50.312+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:50.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:50 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:50.312+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:50.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:50 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:50.312+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:50.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:50 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:50.222+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:50.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:50 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:50.222+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:50.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:50 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:50.222+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:50.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:50 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:50.222+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:51.663 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:51.324+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:51.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:51.324+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:51.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:51.324+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:51.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:51.212+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:51.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:51.212+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:51.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:51.212+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:51.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:51 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:51.212+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:52.663 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:52.353+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:52.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:52.353+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:52.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:52.353+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:52.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:52.224+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:52.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:52.224+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:52.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:52.224+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:52.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:52 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:52.224+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:53.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:53 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:53.374+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.584004+0000 front 2024-09-16T09:18:06.584146+0000 (oldest deadline 2024-09-16T09:18:32.984555+0000) 2024-09-16T09:18:53.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:53 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:53.374+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:12.985368+0000 front 2024-09-16T09:18:12.985226+0000 (oldest deadline 2024-09-16T09:18:37.085176+0000) 2024-09-16T09:18:53.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:53 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:53.374+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:22.386258+0000 front 2024-09-16T09:18:22.386130+0000 (oldest deadline 2024-09-16T09:18:44.085970+0000) 2024-09-16T09:18:53.664 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:53 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6[49088]: 2024-09-16T09:18:53.374+0000 7fbe897a9640 -1 osd.6 765 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-09-16T09:18:29.386829+0000 front 2024-09-16T09:18:29.386853+0000 (oldest deadline 2024-09-16T09:18:52.886745+0000) 2024-09-16T09:18:53.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:53 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:53.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:53.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:53 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:53.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:53.665 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:53 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:53.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:53.665 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:53 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:53.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-09-16T09:18:29.533034+0000 front 2024-09-16T09:18:29.533025+0000 (oldest deadline 2024-09-16T09:18:52.433062+0000) 2024-09-16T09:18:53.665 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:53 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:53.252+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:54.164 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:53 smithi114 podman[68524]: 2024-09-16 09:18:53.903617539 +0000 UTC m=+5.554831562 container died 999f2402213a035b00cf815d495b2834fc19877e533c8f867379dec5826c7290 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-16T09:18:54.504 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:54 smithi114 podman[68524]: 2024-09-16 09:18:54.259838237 +0000 UTC m=+5.911052312 container cleanup 999f2402213a035b00cf815d495b2834fc19877e533c8f867379dec5826c7290 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-09-16T09:18:54.504 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:54 smithi114 bash[68524]: ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6 2024-09-16T09:18:54.504 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:54 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:54.255+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:54.504 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:54 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:54.255+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:54.504 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:54 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:54.255+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:54.504 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:54 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:54.255+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-09-16T09:18:29.533034+0000 front 2024-09-16T09:18:29.533025+0000 (oldest deadline 2024-09-16T09:18:52.433062+0000) 2024-09-16T09:18:54.505 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:54 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:54.255+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:54.821 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:54 smithi114 podman[68546]: 2024-09-16 09:18:54.503223791 +0000 UTC m=+0.595155089 container remove 999f2402213a035b00cf815d495b2834fc19877e533c8f867379dec5826c7290 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-16T09:18:55.164 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:54 smithi114 podman[68639]: 2024-09-16 09:18:54.820694298 +0000 UTC m=+0.111599468 container create ac2b9268379d3fd208f601609b67ab43e484428713702c88f822080360a1400c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-16T09:18:55.164 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:54 smithi114 podman[68639]: 2024-09-16 09:18:54.725034723 +0000 UTC m=+0.015939892 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T09:18:55.164 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:54 smithi114 podman[68639]: 2024-09-16 09:18:54.981746818 +0000 UTC m=+0.272651996 container init ac2b9268379d3fd208f601609b67ab43e484428713702c88f822080360a1400c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-16T09:18:55.164 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:54 smithi114 podman[68639]: 2024-09-16 09:18:54.986752757 +0000 UTC m=+0.277657929 container start ac2b9268379d3fd208f601609b67ab43e484428713702c88f822080360a1400c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-16T09:18:55.164 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:55 smithi114 podman[68639]: 2024-09-16 09:18:55.056252488 +0000 UTC m=+0.347157661 container attach ac2b9268379d3fd208f601609b67ab43e484428713702c88f822080360a1400c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-16T09:18:55.477 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 16 09:18:55 smithi114 podman[68639]: 2024-09-16 09:18:55.184072144 +0000 UTC m=+0.474977313 container died ac2b9268379d3fd208f601609b67ab43e484428713702c88f822080360a1400c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-09-16T09:18:55.477 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:55 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:55.226+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:55.477 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:55 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:55.226+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:55.478 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:55 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:55.226+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:55.478 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:55 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:55.226+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-09-16T09:18:29.533034+0000 front 2024-09-16T09:18:29.533025+0000 (oldest deadline 2024-09-16T09:18:52.433062+0000) 2024-09-16T09:18:55.478 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:55 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:55.226+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:55.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.6.service' 2024-09-16T09:18:55.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:18:55.818 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-16T09:18:55.818 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-16T09:18:55.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.7 2024-09-16T09:18:56.132 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:55 smithi114 systemd[1]: Stopping Ceph osd.7 for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:18:56.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:56 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:56.131+0000 7ff5bfe73640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:18:56.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:56 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:56.131+0000 7ff5bfe73640 -1 osd.7 765 *** Got signal Terminated *** 2024-09-16T09:18:56.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:56 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:56.131+0000 7ff5bfe73640 -1 osd.7 765 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T09:18:56.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:56 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:56.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:56.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:56 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:56.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:56.414 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:56 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:56.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:56.415 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:56 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:56.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-09-16T09:18:29.533034+0000 front 2024-09-16T09:18:29.533025+0000 (oldest deadline 2024-09-16T09:18:52.433062+0000) 2024-09-16T09:18:56.415 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:56 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:56.252+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:57.426 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:18:57 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:18:57.025Z caller=dispatch.go:353 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"\": dial tcp 172.21.15.114:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-09-16T09:18:57.426 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:18:57 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:18:57.025Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-09-16T09:18:57.426 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:18:57 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:18:57.026Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-09-16T09:18:57.588 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:57 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:57.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:57.588 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:57 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:57.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:57.588 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:57 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:57.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:57.588 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:57 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:57.252+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-09-16T09:18:29.533034+0000 front 2024-09-16T09:18:29.533025+0000 (oldest deadline 2024-09-16T09:18:52.433062+0000) 2024-09-16T09:18:57.588 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:57 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:57.252+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:58.541 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:58 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:58.263+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:58.541 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:58 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:58.263+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:58.541 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:58 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:58.263+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:58.541 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:58 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:58.263+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-09-16T09:18:29.533034+0000 front 2024-09-16T09:18:29.533025+0000 (oldest deadline 2024-09-16T09:18:52.433062+0000) 2024-09-16T09:18:58.541 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:58 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:58.263+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:18:59.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:59 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:59.253+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:18:59.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:59 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:59.253+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:18:59.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:59 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:59.253+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:18:59.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:59 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:59.253+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-09-16T09:18:29.533034+0000 front 2024-09-16T09:18:29.533025+0000 (oldest deadline 2024-09-16T09:18:52.433062+0000) 2024-09-16T09:18:59.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:18:59 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:18:59.253+0000 7ff5b8480640 -1 osd.7 765 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14538.0:1147 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e765) 2024-09-16T09:19:00.663 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:00 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:19:00.255+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-09-16T09:18:06.230076+0000 front 2024-09-16T09:18:06.230172+0000 (oldest deadline 2024-09-16T09:18:29.729921+0000) 2024-09-16T09:19:00.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:00 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:19:00.255+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-09-16T09:18:13.131354+0000 front 2024-09-16T09:18:13.131367+0000 (oldest deadline 2024-09-16T09:18:39.031233+0000) 2024-09-16T09:19:00.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:00 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:19:00.255+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-09-16T09:18:19.031852+0000 front 2024-09-16T09:18:19.031828+0000 (oldest deadline 2024-09-16T09:18:44.931748+0000) 2024-09-16T09:19:00.664 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:00 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7[55339]: 2024-09-16T09:19:00.255+0000 7ff5b8480640 -1 osd.7 765 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-09-16T09:18:29.533034+0000 front 2024-09-16T09:18:29.533025+0000 (oldest deadline 2024-09-16T09:18:52.433062+0000) 2024-09-16T09:19:01.838 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:01 smithi114 podman[68766]: 2024-09-16 09:19:01.510847893 +0000 UTC m=+5.473925620 container died d9ab2d4cc32713f86b53ac8a417d9ae0a88b92d9085c351a906faac313faa46f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-16T09:19:01.838 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:01 smithi114 podman[68766]: 2024-09-16 09:19:01.837234956 +0000 UTC m=+5.800312673 container cleanup d9ab2d4cc32713f86b53ac8a417d9ae0a88b92d9085c351a906faac313faa46f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-16T09:19:02.163 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:01 smithi114 bash[68766]: ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7 2024-09-16T09:19:02.164 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:02 smithi114 podman[68799]: 2024-09-16 09:19:02.013299524 +0000 UTC m=+0.495576472 container remove d9ab2d4cc32713f86b53ac8a417d9ae0a88b92d9085c351a906faac313faa46f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True) 2024-09-16T09:19:02.700 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:02 smithi114 podman[68892]: 2024-09-16 09:19:02.4307958 +0000 UTC m=+0.112851829 container create 10eee8b506a5cc1a7e29cfb151a388b91b5d49175a1e6b687cf60dfc5c3105e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73) 2024-09-16T09:19:02.700 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:02 smithi114 podman[68892]: 2024-09-16 09:19:02.333755124 +0000 UTC m=+0.015811159 image pull 425e316ba8ebf74f04aa3640f7deb4fa3f293a78d0c8d097cb679eaf86ddca6c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce 2024-09-16T09:19:02.700 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:02 smithi114 podman[68892]: 2024-09-16 09:19:02.624909688 +0000 UTC m=+0.306965717 container init 10eee8b506a5cc1a7e29cfb151a388b91b5d49175a1e6b687cf60dfc5c3105e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T09:19:02.700 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:02 smithi114 podman[68892]: 2024-09-16 09:19:02.629777797 +0000 UTC m=+0.311833831 container start 10eee8b506a5cc1a7e29cfb151a388b91b5d49175a1e6b687cf60dfc5c3105e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T09:19:02.700 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:02 smithi114 podman[68892]: 2024-09-16 09:19:02.699527297 +0000 UTC m=+0.381583327 container attach 10eee8b506a5cc1a7e29cfb151a388b91b5d49175a1e6b687cf60dfc5c3105e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T09:19:03.105 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:02 smithi114 podman[68892]: 2024-09-16 09:19:02.817843344 +0000 UTC m=+0.499899373 container died 10eee8b506a5cc1a7e29cfb151a388b91b5d49175a1e6b687cf60dfc5c3105e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-16T09:19:03.358 INFO:journalctl@ceph.osd.7.smithi114.stdout:Sep 16 09:19:03 smithi114 podman[68936]: 2024-09-16 09:19:03.335734372 +0000 UTC m=+0.511808824 container remove 10eee8b506a5cc1a7e29cfb151a388b91b5d49175a1e6b687cf60dfc5c3105e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.build-date=20240731) 2024-09-16T09:19:03.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@osd.7.service' 2024-09-16T09:19:03.402 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:19:03.403 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-16T09:19:03.403 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-16T09:19:03.403 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@rgw.foo.a 2024-09-16T09:19:03.710 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:03 smithi111 systemd[1]: Stopping Ceph rgw.foo.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:19:04.176 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a[66889]: 2024-09-16T09:19:03.709+0000 7f5647f42640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T09:19:04.176 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:03 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a[66889]: 2024-09-16T09:19:03.709+0000 7f564b7ae880 -1 shutting down 2024-09-16T09:19:07.427 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:19:07.025Z caller=dispatch.go:353 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"\": dial tcp 172.21.15.111:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-09-16T09:19:07.427 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:19:07.026Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-09-16T09:19:07.427 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:07 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:19:07.026Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-09-16T09:19:14.000 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:13 smithi111 bash[148877]: time="2024-09-16T09:19:13Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-16T09:19:14.001 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:13 smithi111 podman[148877]: 2024-09-16 09:19:13.73066184 +0000 UTC m=+10.135798330 container died ce5ec9cb79b078018b78a0d4ec2fb7e4afd11f763008f33694cc2e84db463e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-16T09:19:14.266 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:14 smithi111 podman[148877]: 2024-09-16 09:19:14.080814323 +0000 UTC m=+10.485950804 container cleanup ce5ec9cb79b078018b78a0d4ec2fb7e4afd11f763008f33694cc2e84db463e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T09:19:14.266 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:14 smithi111 bash[148877]: ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a 2024-09-16T09:19:14.508 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@rgw.foo.a.service' 2024-09-16T09:19:14.530 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:14 smithi111 podman[148913]: 2024-09-16 09:19:14.265266465 +0000 UTC m=+0.527556447 container remove ce5ec9cb79b078018b78a0d4ec2fb7e4afd11f763008f33694cc2e84db463e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-16T09:19:14.531 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:14 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-09-16T09:19:14.531 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:14 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@rgw.foo.a.service: Failed with result 'exit-code'. 2024-09-16T09:19:14.531 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:14 smithi111 systemd[1]: Stopped Ceph rgw.foo.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:19:14.531 INFO:journalctl@ceph.rgw.foo.a.smithi111.stdout:Sep 16 09:19:14 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@rgw.foo.a.service: Consumed 3.824s CPU time. 2024-09-16T09:19:14.545 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:19:14.545 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-16T09:19:14.545 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-16T09:19:14.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-f774c998-7408-11ef-bceb-c7b262605968@prometheus.a 2024-09-16T09:19:14.825 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 systemd[1]: Stopping Ceph prometheus.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:19:15.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.823Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-16T09:19:15.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.824Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-16T09:19:15.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.824Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-16T09:19:15.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.824Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-16T09:19:15.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.824Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-16T09:19:15.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.824Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-16T09:19:15.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.824Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-16T09:19:15.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.824Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-16T09:19:15.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.824Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-16T09:19:15.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.825Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-16T09:19:15.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.825Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-16T09:19:15.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a[64418]: ts=2024-09-16T09:19:14.825Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-16T09:19:15.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:14 smithi114 podman[69020]: 2024-09-16 09:19:14.834148945 +0000 UTC m=+0.093768339 container died 11c080ad8755ea4f8b5700e4bbb8cafbd990fa3fd885bf3623ddb397669163ce (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T09:19:15.414 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:15 smithi114 podman[69020]: 2024-09-16 09:19:15.213093857 +0000 UTC m=+0.472713292 container cleanup 11c080ad8755ea4f8b5700e4bbb8cafbd990fa3fd885bf3623ddb397669163ce (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-16T09:19:15.414 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Sep 16 09:19:15 smithi114 bash[69020]: ceph-f774c998-7408-11ef-bceb-c7b262605968-prometheus-a 2024-09-16T09:19:15.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f774c998-7408-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-16T09:19:15.628 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T09:19:15.628 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-16T09:19:15.629 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f774c998-7408-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T09:19:15.834 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T09:19:17.414 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:17 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:19:17.026Z caller=dispatch.go:353 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"\": dial tcp 172.21.15.111:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-09-16T09:19:17.414 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:17 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:19:17.027Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-09-16T09:19:17.414 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:17 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:19:17.027Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] aggrGroup="{}/{}:{alertname=\"CephNodeDiskspaceWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-09-16T09:19:17.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:17 smithi111 systemd[1]: Stopping Ceph alertmanager.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:19:17.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:17 smithi111 ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a[75737]: ts=2024-09-16T09:19:17.663Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-16T09:19:18.022 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:17 smithi111 podman[149392]: 2024-09-16 09:19:17.704279791 +0000 UTC m=+0.150717340 container died 553e60a063af2e2ccd5d7220ba0fc3073b2d8572974094b4c4eefd5e4e61334b (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T09:19:18.022 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:18 smithi111 podman[149392]: 2024-09-16 09:19:18.021581526 +0000 UTC m=+0.468019080 container cleanup 553e60a063af2e2ccd5d7220ba0fc3073b2d8572974094b4c4eefd5e4e61334b (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T09:19:18.290 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:18 smithi111 bash[149392]: ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a 2024-09-16T09:19:18.291 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:18 smithi111 podman[149409]: 2024-09-16 09:19:18.206271162 +0000 UTC m=+0.498531652 container remove 553e60a063af2e2ccd5d7220ba0fc3073b2d8572974094b4c4eefd5e4e61334b (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-16T09:19:18.291 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:18 smithi111 podman[149409]: 2024-09-16 09:19:18.29006513 +0000 UTC m=+0.582325575 volume remove 4d947a41014c4b82e6ddc9e11c220f40e4b8c0755e98bc67a2b7b3542bcde64f 2024-09-16T09:19:18.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:18 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-16T09:19:18.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:18 smithi111 systemd[1]: Stopped Ceph alertmanager.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:19:18.676 INFO:journalctl@ceph.alertmanager.a.smithi111.stdout:Sep 16 09:19:18 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.839s CPU time. 2024-09-16T09:19:18.956 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:19:18 smithi111 systemd[1]: Stopping Ceph node-exporter.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:19:19.211 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:19:18 smithi111 podman[149529]: 2024-09-16 09:19:18.955928255 +0000 UTC m=+0.132794366 container died 0ffad191150764cf1a630a34475ce5e0a2122c4a257c58d53d5318a715167f49 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T09:19:19.676 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:19:19 smithi111 podman[149529]: 2024-09-16 09:19:19.409450776 +0000 UTC m=+0.586316924 container cleanup 0ffad191150764cf1a630a34475ce5e0a2122c4a257c58d53d5318a715167f49 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T09:19:19.676 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:19:19 smithi111 bash[149529]: ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a 2024-09-16T09:19:19.976 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:19:19 smithi111 podman[149547]: 2024-09-16 09:19:19.693707832 +0000 UTC m=+0.741261730 container remove 0ffad191150764cf1a630a34475ce5e0a2122c4a257c58d53d5318a715167f49 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-16T09:19:19.976 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:19:19 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-16T09:19:19.976 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:19:19 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-16T09:19:19.977 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:19:19 smithi111 systemd[1]: Stopped Ceph node-exporter.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:19:19.977 INFO:journalctl@ceph.node-exporter.a.smithi111.stdout:Sep 16 09:19:19 smithi111 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 3.550s CPU time. 2024-09-16T09:19:20.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f774c998-7408-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T09:19:20.656 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T09:19:22.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:21 smithi114 systemd[1]: Stopping Ceph iscsi.iscsi.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:19:22.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:22 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a[60473]: debug Shutdown received 2024-09-16T09:19:32.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:32 smithi114 bash[69438]: time="2024-09-16T09:19:32Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-16T09:19:32.414 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:32 smithi114 podman[69438]: 2024-09-16 09:19:32.160283405 +0000 UTC m=+10.108032855 container died a0240c3768d6c630cfef2553f101a571da8e7a69ef596227f1e8b0aee3393150 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-16T09:19:32.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:32 smithi114 podman[69438]: 2024-09-16 09:19:32.482749631 +0000 UTC m=+10.430499080 container cleanup a0240c3768d6c630cfef2553f101a571da8e7a69ef596227f1e8b0aee3393150 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T09:19:32.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:32 smithi114 bash[69438]: ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a 2024-09-16T09:19:32.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:32 smithi114 podman[69461]: 2024-09-16 09:19:32.658756707 +0000 UTC m=+0.498664676 container remove a0240c3768d6c630cfef2553f101a571da8e7a69ef596227f1e8b0aee3393150 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f7ad55f9ccd17ba66aadb19b4d291e67b60ea24e5d8a0c0e92941fd59c81e4ce, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-13-1248-8293d73, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T09:19:32.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:32 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-16T09:19:33.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:32 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-16T09:19:33.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:32 smithi114 systemd[1]: Stopped Ceph iscsi.iscsi.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:19:33.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi114.stdout:Sep 16 09:19:32 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Consumed 2.754s CPU time. 2024-09-16T09:19:34.511 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:34 smithi114 systemd[1]: Stopping Ceph grafana.a for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:19:34.512 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:34 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=server t=2024-09-16T09:19:34.509891156Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-16T09:19:34.512 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:34 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=ticker t=2024-09-16T09:19:34.510234638Z level=info msg=stopped last_tick=2024-09-16T09:19:30Z 2024-09-16T09:19:34.512 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:34 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=tracing t=2024-09-16T09:19:34.510266165Z level=info msg="Closing tracing" 2024-09-16T09:19:34.512 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:34 smithi114 ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a[63142]: logger=grafana-apiserver t=2024-09-16T09:19:34.510313695Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-16T09:19:34.775 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:34 smithi114 podman[69743]: 2024-09-16 09:19:34.520330939 +0000 UTC m=+0.107401109 container died b180e6da2fe4041b6cf7974e65a876325f2c0d82ae6109beba933b31baed19f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T09:19:35.034 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:34 smithi114 podman[69743]: 2024-09-16 09:19:34.840500705 +0000 UTC m=+0.427570863 container cleanup b180e6da2fe4041b6cf7974e65a876325f2c0d82ae6109beba933b31baed19f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T09:19:35.034 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:34 smithi114 bash[69743]: ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a 2024-09-16T09:19:35.285 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:35 smithi114 podman[69767]: 2024-09-16 09:19:35.033372151 +0000 UTC m=+0.514042346 container remove b180e6da2fe4041b6cf7974e65a876325f2c0d82ae6109beba933b31baed19f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-16T09:19:35.285 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:35 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-16T09:19:35.285 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:35 smithi114 systemd[1]: Stopped Ceph grafana.a for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:19:35.285 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Sep 16 09:19:35 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@grafana.a.service: Consumed 12.475s CPU time. 2024-09-16T09:19:35.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:19:35 smithi114 systemd[1]: Stopping Ceph node-exporter.b for f774c998-7408-11ef-bceb-c7b262605968... 2024-09-16T09:19:35.969 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:19:35 smithi114 podman[69891]: 2024-09-16 09:19:35.707494608 +0000 UTC m=+0.098518490 container died b37ec837e20effbf4d0fcde451371946b0a6b9d77d7c9fe6ff077587e387d8af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T09:19:35.970 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:19:35 smithi114 podman[69891]: 2024-09-16 09:19:35.969445426 +0000 UTC m=+0.360469296 container cleanup b37ec837e20effbf4d0fcde451371946b0a6b9d77d7c9fe6ff077587e387d8af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T09:19:36.364 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:19:35 smithi114 bash[69891]: ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b 2024-09-16T09:19:36.365 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:19:36 smithi114 podman[69908]: 2024-09-16 09:19:36.137242594 +0000 UTC m=+0.433445067 container remove b37ec837e20effbf4d0fcde451371946b0a6b9d77d7c9fe6ff077587e387d8af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f774c998-7408-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-16T09:19:36.365 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:19:36 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-16T09:19:36.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:19:36 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-16T09:19:36.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:19:36 smithi114 systemd[1]: Stopped Ceph node-exporter.b for f774c998-7408-11ef-bceb-c7b262605968. 2024-09-16T09:19:36.664 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Sep 16 09:19:36 smithi114 systemd[1]: ceph-f774c998-7408-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 3.589s CPU time. 2024-09-16T09:19:37.014 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T09:19:37.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T09:19:37.075 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-16T09:19:37.077 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi/7906989/remote/smithi111/crash 2024-09-16T09:19:37.078 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/crash -- . 2024-09-16T09:19:37.123 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T09:19:37.124 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2024-09-16T09:19:37.126 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi/7906989/remote/smithi114/crash 2024-09-16T09:19:37.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/crash -- . 2024-09-16T09:19:37.160 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/f774c998-7408-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T09:19:37.160 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-09-16T09:19:37.161 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-16T09:19:37.162 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v CEPHADM_STRAY_DAEMON | head -n 1 2024-09-16T09:19:37.198 INFO:tasks.cephadm:Compressing logs... 2024-09-16T09:19:37.198 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T09:19:37.241 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T09:19:37.264 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-16T09:19:37.264 INFO:teuthology.orchestra.run.smithi111.stderr:: No such file or directory 2024-09-16T09:19:37.264 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-16T09:19:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T09:19:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.0.log: /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T09:19:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-16T09:19:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T09:19:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-16T09:19:37.270 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/tcmu-runner.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:: No such file or directory 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/tcmu-runner.log: /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr: 63.1% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/tcmu-runner.log.gz 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr: 79.7% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-16T09:19:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-16T09:19:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-16T09:19:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mon.a.log: 89.2% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T09:19:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-16T09:19:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.log 2024-09-16T09:19:37.273 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mgr.x.log: 87.1% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T09:19:37.277 INFO:teuthology.orchestra.run.smithi114.stderr: 92.5% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T09:19:37.278 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-16T09:19:37.279 INFO:teuthology.orchestra.run.smithi114.stderr: 91.5% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-16T09:19:37.280 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T09:19:37.280 INFO:teuthology.orchestra.run.smithi114.stderr: 91.5% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T09:19:37.281 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.log: 91.5% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T09:19:37.284 INFO:teuthology.orchestra.run.smithi111.stderr: 95.3% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T09:19:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-16T09:19:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log: 92.8% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T09:19:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T09:19:37.295 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T09:19:37.307 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T09:19:37.328 INFO:teuthology.orchestra.run.smithi111.stderr: 94.1% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-16T09:19:37.706 INFO:teuthology.orchestra.run.smithi111.stderr: 91.3% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-16T09:19:38.006 INFO:teuthology.orchestra.run.smithi111.stderr: 92.9% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-16T09:19:38.155 INFO:teuthology.orchestra.run.smithi114.stderr: 92.9% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-16T09:19:39.524 INFO:teuthology.orchestra.run.smithi111.stderr: 92.1% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-16T09:19:43.528 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-16T09:19:43.602 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-16T09:19:43.659 INFO:teuthology.orchestra.run.smithi111.stderr: 93.9% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-16T09:19:44.044 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-16T09:19:44.415 INFO:teuthology.orchestra.run.smithi114.stderr: 94.0% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-16T09:19:44.417 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-16T09:19:44.417 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m7.161s 2024-09-16T09:19:44.417 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m26.468s 2024-09-16T09:19:44.417 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.959s 2024-09-16T09:19:44.450 INFO:teuthology.orchestra.run.smithi111.stderr: 94.0% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-16T09:19:44.465 INFO:teuthology.orchestra.run.smithi111.stderr: 94.0% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-16T09:19:45.073 INFO:teuthology.orchestra.run.smithi111.stderr: 94.0% -- replaced with /var/log/ceph/f774c998-7408-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-16T09:19:45.075 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-09-16T09:19:45.075 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m7.821s 2024-09-16T09:19:45.075 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m30.958s 2024-09-16T09:19:45.075 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m1.081s 2024-09-16T09:19:45.075 INFO:tasks.cephadm:Archiving logs... 2024-09-16T09:19:45.076 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi/7906989/remote/smithi111/log 2024-09-16T09:19:45.076 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T09:19:49.273 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi/7906989/remote/smithi114/log 2024-09-16T09:19:49.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T09:19:53.180 INFO:tasks.cephadm:Removing cluster... 2024-09-16T09:19:53.181 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f774c998-7408-11ef-bceb-c7b262605968 --force 2024-09-16T09:19:53.385 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T09:19:53.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f774c998-7408-11ef-bceb-c7b262605968 --force 2024-09-16T09:19:53.875 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: f774c998-7408-11ef-bceb-c7b262605968 2024-09-16T09:19:54.168 INFO:tasks.cephadm:Removing cephadm ... 2024-09-16T09:19:54.168 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T09:19:54.184 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T09:19:54.199 INFO:tasks.cephadm:Teardown complete 2024-09-16T09:19:54.200 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T09:19:54.209 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T09:19:54.209 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T09:19:54.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T09:19:54.309 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T09:19:54.318 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T09:19:54.318 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T09:19:54.321 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T09:19:54.333 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-09-16T09:19:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-16T09:19:54.337 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T09:19:54.337 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-09-16T09:19:54.337 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv01.front.sepia.ceph.com 3 6 377 53 -99us[ -73us] +/- 64ms 2024-09-16T09:19:54.337 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 55 +172us[ +198us] +/- 84ms 2024-09-16T09:19:54.337 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -42us[ -15us] +/- 75ms 2024-09-16T09:19:54.337 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T09:19:54.339 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T09:19:54.339 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-16T09:19:54.339 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv01.front.sepia.ceph.com 3 6 377 54 -88us[ -72us] +/- 64ms 2024-09-16T09:19:54.339 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 +231us[ +248us] +/- 84ms 2024-09-16T09:19:54.339 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -38us[ -38us] +/- 75ms 2024-09-16T09:19:54.339 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T09:19:54.340 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T09:19:54.347 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T09:19:54.348 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T09:19:54.356 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T09:19:54.452 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T09:19:54.521 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-16T09:19:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T09:19:54.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-16T09:19:54.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T09:19:54.579 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T09:19:54.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T09:19:54.597 INFO:teuthology.task.internal:Duration was 2040.379060 seconds 2024-09-16T09:19:54.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T09:19:54.606 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T09:19:54.606 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T09:19:54.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T09:19:54.645 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T09:19:54.657 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T09:19:54.966 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T09:19:54.967 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-09-16T09:19:54.967 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T09:19:55.027 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-09-16T09:19:55.028 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T09:19:55.051 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T09:19:55.051 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T09:19:55.070 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T09:19:55.117 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T09:19:55.118 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T09:19:55.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T09:19:55.558 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T09:19:55.568 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T09:19:55.568 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T09:19:55.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T09:19:55.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T09:19:55.626 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T09:19:55.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T09:19:55.655 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-09-16T09:19:55.681 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-09-16T09:19:55.705 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T09:19:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T09:19:55.740 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T09:19:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T09:19:55.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T09:19:55.769 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T09:19:55.770 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi/7906989/remote/smithi111 2024-09-16T09:19:55.770 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T09:19:55.836 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-16_07:44:22-orch:cephadm-wip-guits-main-2024-09-13-1248-distro-default-smithi/7906989/remote/smithi114 2024-09-16T09:19:55.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T09:19:55.874 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T09:19:55.874 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T09:19:55.877 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T09:19:55.923 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T09:19:55.935 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T09:19:55.935 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T09:19:55.944 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T09:19:55.944 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T09:19:55.946 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T09:19:55.961 INFO:teuthology.orchestra.run.smithi111.stdout: 265340 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 09:19 /home/ubuntu/cephtest 2024-09-16T09:19:55.977 INFO:teuthology.orchestra.run.smithi114.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 09:19 /home/ubuntu/cephtest 2024-09-16T09:19:55.978 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T09:19:55.988 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T09:19:56.008 INFO:teuthology.run:Summary data: description: orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} duration: 2040.379059791565 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-16T09:19:56.009 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T09:19:56.090 INFO:teuthology.run:pass